1 """Model and Property classes and associated stuff.
3 A model class represents the structure of entities stored in the
4 datastore. Applications define model classes to indicate the
5 structure of their entities, then instantiate those model classes
8 All model classes must inherit (directly or indirectly) from Model.
9 Through the magic of metaclasses, straightforward assignments in the
10 model class definition can be used to declare the model's structure:
13 name = StringProperty()
14 age = IntegerProperty()
16 We can now create a Person entity and write it to the datastore:
18 p = Person(name='Arthur Dent', age=42)
21 The return value from put() is a Key (see the documentation for
22 ndb/key.py), which can be used to retrieve the same entity later:
25 p2 == p # Returns True
27 To update an entity, simple change its attributes and write it back
28 (note that this doesn't change the key):
30 p2.name = 'Arthur Philip Dent'
33 We can also delete an entity (by using the key):
37 The property definitions in the class body tell the system the names
38 and the types of the fields to be stored in the datastore, whether
39 they must be indexed, their default value, and more.
41 Many different Property types exist. Most are indexed by default, the
42 exceptions indicated in the list below:
44 - StringProperty: a short text string, limited to 500 bytes
46 - TextProperty: an unlimited text string; unindexed
48 - BlobProperty: an unlimited byte string; unindexed
50 - IntegerProperty: a 64-bit signed integer
52 - FloatProperty: a double precision floating point number
54 - BooleanProperty: a bool value
56 - DateTimeProperty: a datetime object. Note: App Engine always uses
59 - DateProperty: a date object
61 - TimeProperty: a time object
63 - GeoPtProperty: a geographical location, i.e. (latitude, longitude)
65 - KeyProperty: a datastore Key value, optionally constrained to
66 referring to a specific kind
68 - UserProperty: a User object (for backwards compatibility only)
70 - StructuredProperty: a field that is itself structured like an
71 entity; see below for more details
73 - LocalStructuredProperty: like StructuredProperty but the on-disk
74 representation is an opaque blob; unindexed
76 - ComputedProperty: a property whose value is computed from other
77 properties by a user-defined function. The property value is
78 written to the datastore so that it can be used in queries, but the
79 value from the datastore is not used when the entity is read back
81 - GenericProperty: a property whose type is not constrained; mostly
82 used by the Expando class (see below) but also usable explicitly
84 - JsonProperty: a property whose value is any object that can be
85 serialized using JSON; the value written to the datastore is a JSON
86 representation of that object
88 - PickleProperty: a property whose value is any object that can be
89 serialized using Python's pickle protocol; the value written to the
90 datastore is the pickled representation of that object, using the
91 highest available pickle protocol
93 Most Property classes have similar constructor signatures. They
94 accept several optional keyword arguments:
96 - name=<string>: the name used to store the property value in the
97 datastore. Unlike the following options, this may also be given as
100 - indexed=<bool>: indicates whether the property should be indexed
101 (allowing queries on this property's value)
103 - repeated=<bool>: indicates that this property can have multiple
104 values in the same entity.
106 - required=<bool>: indicates that this property must be given a value
108 - default=<value>: a default value if no explicit value is given
110 - choices=<list of values>: a list or tuple of allowable values
112 - validator=<function>: a general-purpose validation function. It
113 will be called with two arguments (prop, value) and should either
114 return the validated value or raise an exception. It is also
115 allowed for the function to modify the value, but calling it again
116 on the modified value should not modify the value further. (For
117 example: a validator that returns value.strip() or value.lower() is
118 fine, but one that returns value + '$' is not.)
120 - verbose_name=<value>: A human readable name for this property. This
121 human readable name can be used for html form labels.
123 The repeated, required and default options are mutually exclusive: a
124 repeated property cannot be required nor can it specify a default
125 value (the default is always an empty list and an empty list is always
126 an allowed value), and a required property cannot have a default.
128 Some property types have additional arguments. Some property types
129 do not support all options.
131 Repeated properties are always represented as Python lists; if there
132 is only one value, the list has only one element. When a new list is
133 assigned to a repeated property, all elements of the list are
134 validated. Since it is also possible to mutate lists in place,
135 repeated properties are re-validated before they are written to the
138 No validation happens when an entity is read from the datastore;
139 however property values read that have the wrong type (e.g. a string
140 value for an IntegerProperty) are ignored.
142 For non-repeated properties, None is always a possible value, and no
143 validation is called when the value is set to None. However for
144 required properties, writing the entity to the datastore requires
145 the value to be something other than None (and valid).
147 The StructuredProperty is different from most other properties; it
148 lets you define a sub-structure for your entities. The substructure
149 itself is defined using a model class, and the attribute value is an
150 instance of that model class. However it is not stored in the
151 datastore as a separate entity; instead, its attribute values are
152 included in the parent entity using a naming convention (the name of
153 the structured attribute followed by a dot followed by the name of the
154 subattribute). For example:
156 class Address(Model):
157 street = StringProperty()
158 city = StringProperty()
161 name = StringProperty()
162 address = StructuredProperty(Address)
164 p = Person(name='Harry Potter',
165 address=Address(street='4 Privet Drive',
166 city='Little Whinging'))
169 This would write a single 'Person' entity with three attributes (as
170 you could verify using the Datastore Viewer in the Admin Console):
172 name = 'Harry Potter'
173 address.street = '4 Privet Drive'
174 address.city = 'Little Whinging'
176 Structured property types can be nested arbitrarily deep, but in a
177 hierarchy of nested structured property types, only one level can have
178 the repeated flag set. It is fine to have multiple structured
179 properties referencing the same model class.
181 It is also fine to use the same model class both as a top-level entity
182 class and as for a structured property; however queries for the model
183 class will only return the top-level entities.
185 The LocalStructuredProperty works similar to StructuredProperty on the
186 Python side. For example:
188 class Address(Model):
189 street = StringProperty()
190 city = StringProperty()
193 name = StringProperty()
194 address = LocalStructuredProperty(Address)
196 p = Person(name='Harry Potter',
197 address=Address(street='4 Privet Drive',
198 city='Little Whinging'))
201 However the data written to the datastore is different; it writes a
202 'Person' entity with a 'name' attribute as before and a single
203 'address' attribute whose value is a blob which encodes the Address
204 value (using the standard"protocol buffer" encoding).
206 Sometimes the set of properties is not known ahead of time. In such
207 cases you can use the Expando class. This is a Model subclass that
208 creates properties on the fly, both upon assignment and when loading
209 an entity from the datastore. For example:
211 class SuperPerson(Expando):
212 name = StringProperty()
213 superpower = StringProperty()
215 razorgirl = SuperPerson(name='Molly Millions',
216 superpower='bionic eyes, razorblade hands',
217 rasta_name='Steppin\' Razor',
218 alt_name='Sally Shears')
219 elastigirl = SuperPerson(name='Helen Parr',
220 superpower='stretchable body')
221 elastigirl.max_stretch = 30 # Meters
223 You can inspect the properties of an expando instance using the
224 _properties attribute:
226 >>> print razorgirl._properties.keys()
227 ['rasta_name', 'name', 'superpower', 'alt_name']
228 >>> print elastigirl._properties
229 {'max_stretch': GenericProperty('max_stretch'),
230 'name': StringProperty('name'),
231 'superpower': StringProperty('superpower')}
233 Note: this property exists for plain Model instances too; it is just
234 not as interesting for those.
236 The Model class offers basic query support. You can create a Query
237 object by calling the query() class method. Iterating over a Query
238 object returns the entities matching the query one at a time.
240 Query objects are fully described in the docstring for query.py, but
241 there is one handy shortcut that is only available through
242 Model.query(): positional arguments are interpreted as filter
243 expressions which are combined through an AND operator. For example:
245 Person.query(Person.name == 'Harry Potter', Person.age >= 11)
249 Person.query().filter(Person.name == 'Harry Potter', Person.age >= 11)
251 Keyword arguments passed to .query() are passed along to the Query()
254 It is possible to query for field values of stuctured properties. For
257 qry = Person.query(Person.address.city == 'London')
259 A number of top-level functions also live in this module:
261 - transaction() runs a function inside a transaction
262 - get_multi() reads multiple entities at once
263 - put_multi() writes multiple entities at once
264 - delete_multi() deletes multiple entities at once
266 All these have a corresponding *_async() variant as well.
267 The *_multi_async() functions return a list of Futures.
269 And finally these (without async variants):
271 - in_transaction() tests whether you are currently running in a transaction
272 - @transactional decorates functions that should be run in a transaction
274 There are many other interesting features. For example, Model
275 subclasses may define pre-call and post-call hooks for most operations
276 (get, put, delete, allocate_ids), and Property classes may be
277 subclassed to suit various needs. Documentation for writing a
278 Property subclass is in the docstring for the Property class.
281 __author__
= 'guido@google.com (Guido van Rossum)'
284 import cPickle
as pickle
289 from .google_imports
import datastore_errors
290 from .google_imports
import datastore_types
291 from .google_imports
import users
292 from .google_imports
import datastore_query
293 from .google_imports
import datastore_rpc
294 from .google_imports
import entity_pb
298 # NOTE: 'key' is a common local variable name.
299 from . import key
as key_module
300 Key
= key_module
.Key
# For export.
302 # NOTE: Property and Error classes are added later.
303 __all__
= ['Key', 'BlobKey', 'GeoPt', 'Rollback',
304 'Index', 'IndexState', 'IndexProperty',
305 'ModelAdapter', 'ModelAttribute',
306 'ModelKey', 'MetaModel', 'Model', 'Expando',
307 'transaction', 'transaction_async',
308 'in_transaction', 'transactional', 'non_transactional',
309 'get_multi', 'get_multi_async',
310 'put_multi', 'put_multi_async',
311 'delete_multi', 'delete_multi_async',
312 'get_indexes', 'get_indexes_async',
317 BlobKey
= datastore_types
.BlobKey
318 GeoPt
= datastore_types
.GeoPt
320 Rollback
= datastore_errors
.Rollback
323 class KindError(datastore_errors
.BadValueError
):
324 """Raised when an implementation for a kind can't be found.
326 Also raised when the Kind is not an 8-bit string.
330 class InvalidPropertyError(datastore_errors
.Error
):
331 """Raised when a property is not applicable to a given use.
333 For example, a property must exist and be indexed to be used in a query's
334 projection or group by clause.
337 # Mapping for legacy support.
338 BadProjectionError
= InvalidPropertyError
341 class UnprojectedPropertyError(datastore_errors
.Error
):
342 """Raised when getting a property value that's not in the projection."""
345 class ReadonlyPropertyError(datastore_errors
.Error
):
346 """Raised when attempting to set a property value that is read-only."""
349 class ComputedPropertyError(ReadonlyPropertyError
):
350 """Raised when attempting to set a value to or delete a computed property."""
353 # Various imported limits.
354 _MAX_LONG
= key_module
._MAX
_LONG
355 _MAX_STRING_LENGTH
= datastore_types
._MAX
_STRING
_LENGTH
357 # Map index directions to human-readable strings.
359 entity_pb
.Index_Property
.ASCENDING
: 'asc',
360 entity_pb
.Index_Property
.DESCENDING
: 'desc',
363 # Map index states to human-readable strings.
365 entity_pb
.CompositeIndex
.ERROR
: 'error',
366 entity_pb
.CompositeIndex
.DELETED
: 'deleting',
367 entity_pb
.CompositeIndex
.READ_WRITE
: 'serving',
368 entity_pb
.CompositeIndex
.WRITE_ONLY
: 'building',
372 class _NotEqualMixin(object):
373 """Mix-in class that implements __ne__ in terms of __eq__."""
375 def __ne__(self
, other
):
376 """Implement self != other as not(self == other)."""
377 eq
= self
.__eq
__(other
)
378 if eq
is NotImplemented:
379 return NotImplemented
383 class IndexProperty(_NotEqualMixin
):
384 """Immutable object representing a single property in an index."""
387 def __new__(cls
, name
, direction
):
389 obj
= object.__new
__(cls
)
391 obj
.__direction
= direction
396 """The property name being indexed, a string."""
401 """The direction in the index for this property, 'asc' or 'desc'."""
402 return self
.__direction
405 """Return a string representation."""
406 return '%s(name=%r, direction=%r)' % (self
.__class
__.__name
__,
410 def __eq__(self
, other
):
411 """Compare two index properties for equality."""
412 if not isinstance(other
, IndexProperty
):
413 return NotImplemented
414 return self
.name
== other
.name
and self
.direction
== other
.direction
417 return hash((self
.name
, self
.direction
))
420 class Index(_NotEqualMixin
):
421 """Immutable object representing an index."""
424 def __new__(cls
, kind
, properties
, ancestor
):
426 obj
= object.__new
__(cls
)
428 obj
.__properties
= properties
429 obj
.__ancestor
= ancestor
434 """The kind being indexed, a string."""
438 def properties(self
):
439 """A list of PropertyIndex objects giving the properties being indexed."""
440 return self
.__properties
444 """Whether this is an ancestor index, a bool."""
445 return self
.__ancestor
448 """Return a string representation."""
450 parts
.append('kind=%r' % self
.kind
)
451 parts
.append('properties=%r' % self
.properties
)
452 parts
.append('ancestor=%s' % self
.ancestor
)
453 return '%s(%s)' % (self
.__class
__.__name
__, ', '.join(parts
))
455 def __eq__(self
, other
):
456 """Compare two indexes."""
457 if not isinstance(other
, Index
):
458 return NotImplemented
459 return (self
.kind
== other
.kind
and
460 self
.properties
== other
.properties
and
461 self
.ancestor
== other
.ancestor
)
464 return hash((self
.kind
, self
.properties
, self
.ancestor
))
467 class IndexState(_NotEqualMixin
):
468 """Immutable object representing and index and its state."""
471 def __new__(cls
, definition
, state
, id):
473 obj
= object.__new
__(cls
)
474 obj
.__definition
= definition
480 def definition(self
):
481 """An Index object describing the index."""
482 return self
.__definition
486 """The index state, a string.
488 Possible values are 'error', 'deleting', 'serving' or 'building'.
494 """The index ID, an integer."""
498 """Return a string representation."""
500 parts
.append('definition=%r' % self
.definition
)
501 parts
.append('state=%r' % self
.state
)
502 parts
.append('id=%d' % self
.id)
503 return '%s(%s)' % (self
.__class
__.__name
__, ', '.join(parts
))
505 def __eq__(self
, other
):
506 """Compare two index states."""
507 if not isinstance(other
, IndexState
):
508 return NotImplemented
509 return (self
.definition
== other
.definition
and
510 self
.state
== other
.state
and
514 return hash((self
.definition
, self
.state
, self
.id))
517 class ModelAdapter(datastore_rpc
.AbstractAdapter
):
518 """Conversions between 'our' Key and Model classes and protobufs.
520 This is needed to construct a Connection object, which in turn is
521 needed to construct a Context object.
523 See the base class docstring for more info about the signatures.
526 def __init__(self
, default_model
=None):
530 default_model: If an implementation for the kind cannot be found, use
531 this model class. If none is specified, an exception will be thrown
534 self
.default_model
= default_model
537 # Make this a context manager to request setting _orig_pb.
538 # Used in query.py by _MultiQuery.run_to_queue().
543 def __exit__(self
, *unused_args
):
546 def pb_to_key(self
, pb
):
547 return Key(reference
=pb
)
549 def key_to_pb(self
, key
):
550 return key
.reference()
552 def pb_to_entity(self
, pb
):
555 if pb
.key().path().element_size():
556 key
= Key(reference
=pb
.key())
558 modelclass
= Model
._kind
_map
.get(kind
, self
.default_model
)
559 if modelclass
is None:
561 "No model class found for kind '%s'. Did you forget to import it?" %
563 entity
= modelclass
._from
_pb
(pb
, key
=key
, set_key
=False)
568 def entity_to_pb(self
, ent
):
572 def pb_to_index(self
, pb
):
573 index_def
= pb
.definition()
574 properties
= [IndexProperty(name
=prop
.name(),
575 direction
=_DIR_MAP
[prop
.direction()])
576 for prop
in index_def
.property_list()]
577 index
= Index(kind
=index_def
.entity_type(),
578 properties
=properties
,
579 ancestor
=bool(index_def
.ancestor()),
581 index_state
= IndexState(definition
=index
,
582 state
=_STATE_MAP
[pb
.state()],
588 def make_connection(config
=None, default_model
=None):
589 """Create a new Connection object with the right adapter.
591 Optionally you can pass in a datastore_rpc.Configuration object.
593 return datastore_rpc
.Connection(
594 adapter
=ModelAdapter(default_model
),
598 class ModelAttribute(object):
599 """A Base class signifying the presence of a _fix_up() method."""
601 def _fix_up(self
, cls
, code_name
):
605 class _BaseValue(_NotEqualMixin
):
606 """A marker object wrapping a 'base type' value.
608 This is used to be able to tell whether ent._values[name] is a
609 user value (i.e. of a type that the Python code understands) or a
610 base value (i.e of a type that serialization understands).
611 User values are unwrapped; base values are wrapped in a
615 __slots__
= ['b_val']
617 def __init__(self
, b_val
):
618 """Constructor. Argument is the base value to be wrapped."""
619 assert b_val
is not None
620 assert not isinstance(b_val
, list), repr(b_val
)
624 return '_BaseValue(%r)' % (self
.b_val
,)
626 def __eq__(self
, other
):
627 if not isinstance(other
, _BaseValue
):
628 return NotImplemented
629 return self
.b_val
== other
.b_val
632 raise TypeError('_BaseValue is not immutable')
635 class Property(ModelAttribute
):
636 """A class describing a typed, persisted attribute of a datastore entity.
638 Not to be confused with Python's 'property' built-in.
640 This is just a base class; there are specific subclasses that
641 describe Properties of various types (and GenericProperty which
642 describes a dynamically typed Property).
644 All special Property attributes, even those considered 'public',
645 have names starting with an underscore, because StructuredProperty
646 uses the non-underscore attribute namespace to refer to nested
647 Property names; this is essential for specifying queries on
648 subproperties (see the module docstring).
650 The Property class and its predefined subclasses allow easy
651 subclassing using composable (or stackable) validation and
652 conversion APIs. These require some terminology definitions:
654 - A 'user value' is a value such as would be set and accessed by the
655 application code using standard attributes on the entity.
657 - A 'base value' is a value such as would be serialized to
658 and deserialized from the datastore.
660 The values stored in ent._values[name] and accessed by
661 _store_value() and _retrieve_value() can be either user values or
662 base values. To retrieve user values, use
663 _get_user_value(). To retrieve base values, use
664 _get_base_value(). In particular, _get_value() calls
665 _get_user_value(), and _serialize() effectively calls
668 To store a user value, just call _store_value(). To store a
669 base value, wrap the value in a _BaseValue() and then
672 A Property subclass that wants to implement a specific
673 transformation between user values and serialiazble values should
674 implement two methods, _to_base_type() and _from_base_type().
675 These should *NOT* call their super() method; super calls are taken
676 care of by _call_to_base_type() and _call_from_base_type().
677 This is what is meant by composable (or stackable) APIs.
679 The API supports 'stacking' classes with ever more sophisticated
680 user<-->base conversions: the user-->base conversion
681 goes from more sophisticated to less sophisticated, while the
682 base-->user conversion goes from less sophisticated to more
683 sophisticated. For example, see the relationship between
684 BlobProperty, TextProperty and StringProperty.
686 In addition to _to_base_type() and _from_base_type(), the
687 _validate() method is also a composable API.
689 The validation API distinguishes between 'lax' and 'strict' user
690 values. The set of lax values is a superset of the set of strict
691 values. The _validate() method takes a lax value and if necessary
692 converts it to a strict value. This means that when setting the
693 property value, lax values are accepted, while when getting the
694 property value, only strict values will be returned. If no
695 conversion is needed, _validate() may return None. If the argument
696 is outside the set of accepted lax values, _validate() should raise
697 an exception, preferably TypeError or
698 datastore_errors.BadValueError.
702 def _validate(self, value):
703 'Lax user value to strict user value.'
704 if not isinstance(value, <top type>):
705 raise TypeError(...) # Or datastore_errors.BadValueError(...).
707 def _to_base_type(self, value):
708 '(Strict) user value to base value.'
709 if isinstance(value, <user type>):
710 return <base type>(value)
712 def _from_base_type(self, value):
713 'base value to (strict) user value.'
714 if not isinstance(value, <base type>):
715 return <user type>(value)
717 Things that _validate(), _to_base_type() and _from_base_type()
718 do *not* need to handle:
720 - None: They will not be called with None (and if they return None,
721 this means that the value does not need conversion).
723 - Repeated values: The infrastructure (_get_user_value() and
724 _get_base_value()) takes care of calling
725 _from_base_type() or _to_base_type() for each list item in a
728 - Wrapping values in _BaseValue(): The wrapping and unwrapping is
729 taken care of by the infrastructure that calls the composable APIs.
731 - Comparisons: The comparison operations call _to_base_type() on
734 - Distinguishing between user and base values: the
735 infrastructure guarantees that _from_base_type() will be called
736 with an (unwrapped) base value, and that
737 _to_base_type() will be called with a user value.
739 - Returning the original value: if any of these return None, the
740 original value is kept. (Returning a differen value not equal to
741 None will substitute the different value.)
744 # TODO: Separate 'simple' properties from base Property class
756 __creation_counter_global
= 0
758 _attributes
= ['_name', '_indexed', '_repeated', '_required', '_default',
759 '_choices', '_validator', '_verbose_name']
760 _positional
= 1 # Only name is a positional argument.
762 @utils.positional(1 + _positional
) # Add 1 for self.
763 def __init__(self
, name
=None, indexed
=None, repeated
=None,
764 required
=None, default
=None, choices
=None, validator
=None,
766 """Constructor. For arguments see the module docstring."""
768 if isinstance(name
, unicode):
769 name
= name
.encode('utf-8')
770 if not isinstance(name
, str):
771 raise TypeError('Name %r is not a string' % (name
,))
773 raise ValueError('Name %r cannot contain period characters' % (name
,))
775 if indexed
is not None:
776 self
._indexed
= indexed
777 if repeated
is not None:
778 self
._repeated
= repeated
779 if required
is not None:
780 self
._required
= required
781 if default
is not None:
782 # TODO: Call _validate() on default?
783 self
._default
= default
784 if verbose_name
is not None:
785 self
._verbose
_name
= verbose_name
786 if (bool(self
._repeated
) +
787 bool(self
._required
) +
788 (self
._default
is not None)) > 1:
789 raise ValueError('repeated, required and default are mutally exclusive.')
790 if choices
is not None:
791 if not isinstance(choices
, (list, tuple, set, frozenset)):
792 raise TypeError('choices must be a list, tuple or set; received %r' %
794 # TODO: Call _validate() on each choice?
795 self
._choices
= frozenset(choices
)
796 if validator
is not None:
797 # The validator is called as follows:
798 # value = validator(prop, value)
799 # It should return the value to be used, or raise an exception.
800 # It should be idempotent, i.e. calling it a second time should
801 # not further modify the value. So a validator that returns e.g.
802 # value.lower() or value.strip() is fine, but one that returns
803 # value + '$' is not.
804 if not hasattr(validator
, '__call__'):
805 raise TypeError('validator must be callable or None; received %r' %
807 self
._validator
= validator
808 # Keep a unique creation counter.
809 Property
.__creation
_counter
_global
+= 1
810 self
._creation
_counter
= Property
.__creation
_counter
_global
813 """Return a compact unambiguous string representation of a property."""
816 for i
, attr
in enumerate(self
._attributes
):
817 val
= getattr(self
, attr
)
818 if val
is not getattr(cls
, attr
):
819 if isinstance(val
, type):
823 if i
>= cls
._positional
:
824 if attr
.startswith('_'):
826 s
= '%s=%s' % (attr
, s
)
828 s
= '%s(%s)' % (self
.__class
__.__name
__, ', '.join(args
))
831 def _datastore_type(self
, value
):
832 """Internal hook used by property filters.
834 Sometimes the low-level query interface needs a specific data type
835 in order for the right filter to be constructed. See _comparison().
839 def _comparison(self
, op
, value
):
840 """Internal helper for comparison operators.
843 op: The operator ('=', '<' etc.).
846 A FilterNode instance representing the requested comparison.
848 # NOTE: This is also used by query.gql().
849 if not self
._indexed
:
850 raise datastore_errors
.BadFilterError(
851 'Cannot query for unindexed property %s' % self
._name
)
852 from .query
import FilterNode
# Import late to avoid circular imports.
853 if value
is not None:
854 value
= self
._do
_validate
(value
)
855 value
= self
._call
_to
_base
_type
(value
)
856 value
= self
._datastore
_type
(value
)
857 return FilterNode(self
._name
, op
, value
)
859 # Comparison operators on Property instances don't compare the
860 # properties; instead they return FilterNode instances that can be
861 # used in queries. See the module docstrings above and in query.py
862 # for details on how these can be used.
864 def __eq__(self
, value
):
865 """Return a FilterNode instance representing the '=' comparison."""
866 return self
._comparison
('=', value
)
868 def __ne__(self
, value
):
869 """Return a FilterNode instance representing the '!=' comparison."""
870 return self
._comparison
('!=', value
)
872 def __lt__(self
, value
):
873 """Return a FilterNode instance representing the '<' comparison."""
874 return self
._comparison
('<', value
)
876 def __le__(self
, value
):
877 """Return a FilterNode instance representing the '<=' comparison."""
878 return self
._comparison
('<=', value
)
880 def __gt__(self
, value
):
881 """Return a FilterNode instance representing the '>' comparison."""
882 return self
._comparison
('>', value
)
884 def __ge__(self
, value
):
885 """Return a FilterNode instance representing the '>=' comparison."""
886 return self
._comparison
('>=', value
)
888 def _IN(self
, value
):
889 """Comparison operator for the 'in' comparison operator.
891 The Python 'in' operator cannot be overloaded in the way we want
892 to, so we define a method. For example:
894 Employee.query(Employee.rank.IN([4, 5, 6]))
896 Note that the method is called ._IN() but may normally be invoked
897 as .IN(); ._IN() is provided for the case you have a
898 StructuredProperty with a model that has a Property named IN.
900 if not self
._indexed
:
901 raise datastore_errors
.BadFilterError(
902 'Cannot query for unindexed property %s' % self
._name
)
903 from .query
import FilterNode
# Import late to avoid circular imports.
904 if not isinstance(value
, (list, tuple, set, frozenset)):
905 raise datastore_errors
.BadArgumentError(
906 'Expected list, tuple or set, got %r' % (value
,))
910 val
= self
._do
_validate
(val
)
911 val
= self
._call
_to
_base
_type
(val
)
912 val
= self
._datastore
_type
(val
)
914 return FilterNode(self
._name
, 'in', values
)
918 """Return a descending sort order on this Property.
922 Employee.query().order(-Employee.rank)
924 return datastore_query
.PropertyOrder(
925 self
._name
, datastore_query
.PropertyOrder
.DESCENDING
)
928 """Return an ascending sort order on this Property.
930 Note that this is redundant but provided for consistency with
931 __neg__. For example, the following two are equivalent:
933 Employee.query().order(+Employee.rank)
934 Employee.query().order(Employee.rank)
936 return datastore_query
.PropertyOrder(self
._name
)
938 def _do_validate(self
, value
):
939 """Call all validations on the value.
941 This calls the most derived _validate() method(s), then the custom
942 validator function, and then checks the choices. It returns the
943 value, possibly modified in an idempotent way, or raises an
946 Note that this does not call all composable _validate() methods.
947 It only calls _validate() methods up to but not including the
948 first _to_base_type() method, when the MRO is traversed looking
949 for _validate() and _to_base_type() methods. (IOW if a class
950 defines both _validate() and _to_base_type(), its _validate()
951 is called and then the search is aborted.)
953 Note that for a repeated Property this function should be called
954 for each item in the list, not for the list as a whole.
956 if isinstance(value
, _BaseValue
):
958 value
= self
._call
_shallow
_validation
(value
)
959 if self
._validator
is not None:
960 newvalue
= self
._validator
(self
, value
)
961 if newvalue
is not None:
963 if self
._choices
is not None:
964 if value
not in self
._choices
:
965 raise datastore_errors
.BadValueError(
966 'Value %r for property %s is not an allowed choice' %
970 def _fix_up(self
, cls
, code_name
):
971 """Internal helper called to tell the property its name.
973 This is called by _fix_up_properties() which is called by
974 MetaModel when finishing the construction of a Model subclass.
975 The name passed in is the name of the class attribute to which the
976 Property is assigned (a.k.a. the code name). Note that this means
977 that each Property instance must be assigned to (at most) one
978 class attribute. E.g. to declare three strings, you must call
979 StringProperty() three times, you cannot write
981 foo = bar = baz = StringProperty()
983 self
._code
_name
= code_name
984 if self
._name
is None:
985 self
._name
= code_name
987 def _store_value(self
, entity
, value
):
988 """Internal helper to store a value in an entity for a Property.
990 This assumes validation has already taken place. For a repeated
991 Property the value should be a list.
993 entity
._values
[self
._name
] = value
995 def _set_value(self
, entity
, value
):
996 """Internal helper to set a value in an entity for a Property.
998 This performs validation first. For a repeated Property the value
1001 if entity
._projection
:
1002 raise ReadonlyPropertyError(
1003 'You cannot set property values of a projection entity')
1005 if not isinstance(value
, (list, tuple, set, frozenset)):
1006 raise datastore_errors
.BadValueError('Expected list or tuple, got %r' %
1008 value
= [self
._do
_validate
(v
) for v
in value
]
1010 if value
is not None:
1011 value
= self
._do
_validate
(value
)
1012 self
._store
_value
(entity
, value
)
1014 def _has_value(self
, entity
, unused_rest
=None):
1015 """Internal helper to ask if the entity has a value for this Property."""
1016 return self
._name
in entity
._values
1018 def _retrieve_value(self
, entity
, default
=None):
1019 """Internal helper to retrieve the value for this Property from an entity.
1021 This returns None if no value is set, or the default argument if
1022 given. For a repeated Property this returns a list if a value is
1023 set, otherwise None. No additional transformations are applied.
1025 return entity
._values
.get(self
._name
, default
)
1027 def _get_user_value(self
, entity
):
1028 """Return the user value for this property of the given entity.
1030 This implies removing the _BaseValue() wrapper if present, and
1031 if it is, calling all _from_base_type() methods, in the reverse
1032 method resolution order of the property's class. It also handles
1033 default values and repeated properties.
1035 return self
._apply
_to
_values
(entity
, self
._opt
_call
_from
_base
_type
)
1037 def _get_base_value(self
, entity
):
1038 """Return the base value for this property of the given entity.
1040 This implies calling all _to_base_type() methods, in the method
1041 resolution order of the property's class, and adding a
1042 _BaseValue() wrapper, if one is not already present. (If one
1043 is present, no work is done.) It also handles default values and
1044 repeated properties.
1046 return self
._apply
_to
_values
(entity
, self
._opt
_call
_to
_base
_type
)
1048 # TODO: Invent a shorter name for this.
1049 def _get_base_value_unwrapped_as_list(self
, entity
):
1050 """Like _get_base_value(), but always returns a list.
1053 A new list of unwrapped base values. For an unrepeated
1054 property, if the value is missing or None, returns [None]; for a
1055 repeated property, if the original value is missing or None or
1058 wrapped
= self
._get
_base
_value
(entity
)
1062 assert isinstance(wrapped
, list)
1063 return [w
.b_val
for w
in wrapped
]
1067 assert isinstance(wrapped
, _BaseValue
)
1068 return [wrapped
.b_val
]
1070 def _opt_call_from_base_type(self
, value
):
1071 """Call _from_base_type() if necessary.
1073 If the value is a _BaseValue instance, unwrap it and call all
1074 _from_base_type() methods. Otherwise, return the value
1077 if isinstance(value
, _BaseValue
):
1078 value
= self
._call
_from
_base
_type
(value
.b_val
)
1081 def _value_to_repr(self
, value
):
1082 """Turn a value (base or not) into its repr().
1084 This exists so that property classes can override it separately.
1086 # Manually apply _from_base_type() so as not to have a side
1087 # effect on what's contained in the entity. Printing a value
1088 # should not change it!
1089 val
= self
._opt
_call
_from
_base
_type
(value
)
1092 def _opt_call_to_base_type(self
, value
):
1093 """Call _to_base_type() if necessary.
1095 If the value is a _BaseValue instance, return it unchanged.
1096 Otherwise, call all _validate() and _to_base_type() methods and
1097 wrap it in a _BaseValue instance.
1099 if not isinstance(value
, _BaseValue
):
1100 value
= _BaseValue(self
._call
_to
_base
_type
(value
))
1103 def _call_from_base_type(self
, value
):
1104 """Call all _from_base_type() methods on the value.
1106 This calls the methods in the reverse method resolution order of
1107 the property's class.
1109 methods
= self
._find
_methods
('_from_base_type', reverse
=True)
1110 call
= self
._apply
_list
(methods
)
1113 def _call_to_base_type(self
, value
):
1114 """Call all _validate() and _to_base_type() methods on the value.
1116 This calls the methods in the method resolution order of the
1119 methods
= self
._find
_methods
('_validate', '_to_base_type')
1120 call
= self
._apply
_list
(methods
)
1123 def _call_shallow_validation(self
, value
):
1124 """Call the initial set of _validate() methods.
1126 This is similar to _call_to_base_type() except it only calls
1127 those _validate() methods that can be called without needing to
1128 call _to_base_type().
1130 An example: suppose the class hierarchy is A -> B -> C ->
1131 Property, and suppose A defines _validate() only, but B and C
1132 define _validate() and _to_base_type(). The full list of
1133 methods called by _call_to_base_type() is:
1141 This method will call A._validate() and B._validate() but not the
1145 for method
in self
._find
_methods
('_validate', '_to_base_type'):
1146 if method
.__name
__ != '_validate':
1148 methods
.append(method
)
1149 call
= self
._apply
_list
(methods
)
1153 def _find_methods(cls
, *names
, **kwds
):
1154 """Compute a list of composable methods.
1156 Because this is a common operation and the class hierarchy is
1157 static, the outcome is cached (assuming that for a particular list
1158 of names the reversed flag is either always on, or always off).
1161 *names: One or more method names.
1162 reverse: Optional flag, default False; if True, the list is
1166 A list of callable class method objects.
1168 reverse
= kwds
.pop('reverse', False)
1169 assert not kwds
, repr(kwds
)
1170 cache
= cls
.__dict
__.get('_find_methods_cache')
1172 hit
= cache
.get(names
)
1176 cls
._find
_methods
_cache
= cache
= {}
1178 for c
in cls
.__mro
__:
1180 method
= c
.__dict
__.get(name
)
1181 if method
is not None:
1182 methods
.append(method
)
1185 cache
[names
] = methods
1188 def _apply_list(self
, methods
):
1189 """Return a single callable that applies a list of methods to a value.
1191 If a method returns None, the last value is kept; if it returns
1192 some other value, that replaces the last value. Exceptions are
1196 for method
in methods
:
1197 newvalue
= method(self
, value
)
1198 if newvalue
is not None:
1203 def _apply_to_values(self
, entity
, function
):
1204 """Apply a function to the property value/values of a given entity.
1206 This retrieves the property value, applies the function, and then
1207 stores the value back. For a repeated property, the function is
1208 applied separately to each of the values in the list. The
1209 resulting value or list of values is both stored back in the
1210 entity and returned from this method.
1212 value
= self
._retrieve
_value
(entity
, self
._default
)
1216 self
._store
_value
(entity
, value
)
1218 value
[:] = map(function
, value
)
1220 if value
is not None:
1221 newvalue
= function(value
)
1222 if newvalue
is not None and newvalue
is not value
:
1223 self
._store
_value
(entity
, newvalue
)
1227 def _get_value(self
, entity
):
1228 """Internal helper to get the value for this Property from an entity.
1230 For a repeated Property this initializes the value to an empty
1231 list if it is not set.
1233 if entity
._projection
:
1234 if self
._name
not in entity
._projection
:
1235 raise UnprojectedPropertyError(
1236 'Property %s is not in the projection' % (self
._name
,))
1237 return self
._get
_user
_value
(entity
)
1239 def _delete_value(self
, entity
):
1240 """Internal helper to delete the value for this Property from an entity.
1242 Note that if no value exists this is a no-op; deleted values will
1243 not be serialized but requesting their value will return None (or
1244 an empty list in the case of a repeated Property).
1246 if self
._name
in entity
._values
:
1247 del entity
._values
[self
._name
]
1249 def _is_initialized(self
, entity
):
1250 """Internal helper to ask if the entity has a value for this Property.
1252 This returns False if a value is stored but it is None.
1254 return not self
._required
or (self
._has
_value
(entity
) and
1255 self
._get
_value
(entity
) is not None)
1257 def __get__(self
, entity
, unused_cls
=None):
1258 """Descriptor protocol: get the value from the entity."""
1260 return self
# __get__ called on class
1261 return self
._get
_value
(entity
)
1263 def __set__(self
, entity
, value
):
1264 """Descriptor protocol: set the value on the entity."""
1265 self
._set
_value
(entity
, value
)
1267 def __delete__(self
, entity
):
1268 """Descriptor protocol: delete the value from the entity."""
1269 self
._delete
_value
(entity
)
1271 def _serialize(self
, entity
, pb
, prefix
='', parent_repeated
=False,
1273 """Internal helper to serialize this property to a protocol buffer.
1275 Subclasses may override this method.
1278 entity: The entity, a Model (subclass) instance.
1279 pb: The protocol buffer, an EntityProto instance.
1280 prefix: Optional name prefix used for StructuredProperty
1281 (if present, must end in '.').
1282 parent_repeated: True if the parent (or an earlier ancestor)
1283 is a repeated Property.
1284 projection: A list or tuple of strings representing the projection for
1285 the model instance, or None if the instance is not a projection.
1287 values
= self
._get
_base
_value
_unwrapped
_as
_list
(entity
)
1289 name
= prefix
+ self
._name
1290 if projection
and name
not in projection
:
1293 p
= pb
.add_property()
1295 p
= pb
.add_raw_property()
1297 p
.set_multiple(self
._repeated
or parent_repeated
)
1298 v
= p
.mutable_value()
1300 self
._db
_set
_value
(v
, p
, val
)
1302 # Projected properties have the INDEX_VALUE meaning and only contain
1303 # the original property's name and value.
1304 new_p
= entity_pb
.Property()
1305 new_p
.set_name(p
.name())
1306 new_p
.set_meaning(entity_pb
.Property
.INDEX_VALUE
)
1307 new_p
.set_multiple(False)
1308 new_p
.mutable_value().CopyFrom(v
)
1311 def _deserialize(self
, entity
, p
, unused_depth
=1):
1312 """Internal helper to deserialize this property from a protocol buffer.
1314 Subclasses may override this method.
1317 entity: The entity, a Model (subclass) instance.
1318 p: A Property Message object (a protocol buffer).
1319 depth: Optional nesting depth, default 1 (unused here, but used
1320 by some subclasses that override this method).
1323 val
= self
._db
_get
_value
(v
, p
)
1325 val
= _BaseValue(val
)
1327 if self
._has
_value
(entity
):
1328 value
= self
._retrieve
_value
(entity
)
1329 assert isinstance(value
, list), repr(value
)
1335 self
._store
_value
(entity
, value
)
1337 def _prepare_for_put(self
, entity
):
1340 def _check_property(self
, rest
=None, require_indexed
=True):
1341 """Internal helper to check this property for specific requirements.
1343 Called by Model._check_properties().
1346 rest: Optional subproperty to check, of the form 'name1.name2...nameN'.
1349 InvalidPropertyError if this property does not meet the given
1350 requirements or if a subproperty is specified. (StructuredProperty
1351 overrides this method to handle subproperties.)
1353 if require_indexed
and not self
._indexed
:
1354 raise InvalidPropertyError('Property is unindexed %s' % self
._name
)
1356 raise InvalidPropertyError('Referencing subproperty %s.%s '
1357 'but %s is not a structured property' %
1358 (self
._name
, rest
, self
._name
))
1360 def _get_for_dict(self
, entity
):
1361 """Retrieve the value like _get_value(), processed for _to_dict().
1363 Property subclasses can override this if they want the dictionary
1364 returned by entity._to_dict() to contain a different value. The
1365 main use case is StructuredProperty and LocalStructuredProperty.
1369 - If you override _get_for_dict() to return a different type, you
1370 must override _validate() to accept values of that type and
1371 convert them back to the original type.
1373 - If you override _get_for_dict(), you must handle repeated values
1374 and None correctly. (See _StructuredGetForDictMixin for an
1375 example.) However, _validate() does not need to handle these.
1377 return self
._get
_value
(entity
)
1380 def _validate_key(value
, entity
=None):
1381 if not isinstance(value
, Key
):
1382 # TODO: BadKeyError.
1383 raise datastore_errors
.BadValueError('Expected Key, got %r' % value
)
1384 if entity
and entity
.__class
__ not in (Model
, Expando
):
1385 if value
.kind() != entity
._get
_kind
():
1386 raise KindError('Expected Key kind to be %s; received %s' %
1387 (entity
._get
_kind
(), value
.kind()))
1391 class ModelKey(Property
):
1392 """Special property to store the Model key."""
1395 super(ModelKey
, self
).__init
__()
1396 self
._name
= '__key__'
1398 def _datastore_type(self
, value
):
1399 return datastore_types
.Key(value
.urlsafe())
1401 def _comparison(self
, op
, value
):
1402 if value
is not None:
1403 return super(ModelKey
, self
)._comparison
(op
, value
)
1404 raise datastore_errors
.BadValueError(
1405 "__key__ filter query can't be compared to None")
1407 # TODO: Support IN().
1409 def _validate(self
, value
):
1410 return _validate_key(value
)
1412 def _set_value(self
, entity
, value
):
1413 """Setter for key attribute."""
1414 if value
is not None:
1415 value
= _validate_key(value
, entity
=entity
)
1416 value
= entity
._validate
_key
(value
)
1417 entity
._entity
_key
= value
1419 def _get_value(self
, entity
):
1420 """Getter for key attribute."""
1421 return entity
._entity
_key
1423 def _delete_value(self
, entity
):
1424 """Deleter for key attribute."""
1425 entity
._entity
_key
= None
1428 class BooleanProperty(Property
):
1429 """A Property whose value is a Python bool."""
1430 # TODO: Allow int/long values equal to 0 or 1?
1432 def _validate(self
, value
):
1433 if not isinstance(value
, bool):
1434 raise datastore_errors
.BadValueError('Expected bool, got %r' %
1438 def _db_set_value(self
, v
, unused_p
, value
):
1439 if not isinstance(value
, bool):
1440 raise TypeError('BooleanProperty %s can only be set to bool values; '
1441 'received %r' % (self
._name
, value
))
1442 v
.set_booleanvalue(value
)
1444 def _db_get_value(self
, v
, unused_p
):
1445 if not v
.has_booleanvalue():
1447 # The booleanvalue field is an int32, so booleanvalue() returns an
1448 # int, hence the conversion.
1449 return bool(v
.booleanvalue())
1452 class IntegerProperty(Property
):
1453 """A Property whose value is a Python int or long (or bool)."""
1455 def _validate(self
, value
):
1456 if not isinstance(value
, (int, long)):
1457 raise datastore_errors
.BadValueError('Expected integer, got %r' %
1461 def _db_set_value(self
, v
, unused_p
, value
):
1462 if not isinstance(value
, (bool, int, long)):
1463 raise TypeError('IntegerProperty %s can only be set to integer values; '
1464 'received %r' % (self
._name
, value
))
1465 v
.set_int64value(value
)
1467 def _db_get_value(self
, v
, unused_p
):
1468 if not v
.has_int64value():
1470 return int(v
.int64value())
1473 class FloatProperty(Property
):
1474 """A Property whose value is a Python float.
1476 Note: int, long and bool are also allowed.
1479 def _validate(self
, value
):
1480 if not isinstance(value
, (int, long, float)):
1481 raise datastore_errors
.BadValueError('Expected float, got %r' %
1485 def _db_set_value(self
, v
, unused_p
, value
):
1486 if not isinstance(value
, (bool, int, long, float)):
1487 raise TypeError('FloatProperty %s can only be set to integer or float '
1488 'values; received %r' % (self
._name
, value
))
1489 v
.set_doublevalue(float(value
))
1491 def _db_get_value(self
, v
, unused_p
):
1492 if not v
.has_doublevalue():
1494 return v
.doublevalue()
1497 # A custom 'meaning' for compressed properties.
1498 _MEANING_URI_COMPRESSED
= 'ZLIB'
1501 class _CompressedValue(_NotEqualMixin
):
1502 """A marker object wrapping compressed values."""
1504 __slots__
= ['z_val']
1506 def __init__(self
, z_val
):
1507 """Constructor. Argument is a string returned by zlib.compress()."""
1508 assert isinstance(z_val
, str), repr(z_val
)
1512 return '_CompressedValue(%s)' % repr(self
.z_val
)
1514 def __eq__(self
, other
):
1515 if not isinstance(other
, _CompressedValue
):
1516 return NotImplemented
1517 return self
.z_val
== other
.z_val
1520 raise TypeError('_CompressedValue is not immutable')
1523 class BlobProperty(Property
):
1524 """A Property whose value is a byte string. It may be compressed."""
1529 _attributes
= Property
._attributes
+ ['_compressed']
1531 @utils.positional(1 + Property
._positional
)
1532 def __init__(self
, name
=None, compressed
=False, **kwds
):
1533 super(BlobProperty
, self
).__init
__(name
=name
, **kwds
)
1534 self
._compressed
= compressed
1535 if compressed
and self
._indexed
:
1536 # TODO: Allow this, but only allow == and IN comparisons?
1537 raise NotImplementedError('BlobProperty %s cannot be compressed and '
1538 'indexed at the same time.' % self
._name
)
1540 def _value_to_repr(self
, value
):
1541 long_repr
= super(BlobProperty
, self
)._value
_to
_repr
(value
)
1542 # Note that we may truncate even if the value is shorter than
1543 # _MAX_STRING_LENGTH; e.g. if it contains many \xXX or \uUUUU
1545 if len(long_repr
) > _MAX_STRING_LENGTH
+ 4:
1546 # Truncate, assuming the final character is the closing quote.
1547 long_repr
= long_repr
[:_MAX_STRING_LENGTH
] + '...' + long_repr
[-1]
1550 def _validate(self
, value
):
1551 if not isinstance(value
, str):
1552 raise datastore_errors
.BadValueError('Expected str, got %r' %
1554 if (self
._indexed
and
1555 not isinstance(self
, TextProperty
) and
1556 len(value
) > _MAX_STRING_LENGTH
):
1557 raise datastore_errors
.BadValueError(
1558 'Indexed value %s must be at most %d bytes' %
1559 (self
._name
, _MAX_STRING_LENGTH
))
1561 def _to_base_type(self
, value
):
1562 if self
._compressed
:
1563 return _CompressedValue(zlib
.compress(value
))
1565 def _from_base_type(self
, value
):
1566 if isinstance(value
, _CompressedValue
):
1567 return zlib
.decompress(value
.z_val
)
1569 def _datastore_type(self
, value
):
1570 # Since this is only used for queries, and queries imply an
1571 # indexed property, always use ByteString.
1572 return datastore_types
.ByteString(value
)
1574 def _db_set_value(self
, v
, p
, value
):
1575 if isinstance(value
, _CompressedValue
):
1576 self
._db
_set
_compressed
_meaning
(p
)
1579 self
._db
_set
_uncompressed
_meaning
(p
)
1580 v
.set_stringvalue(value
)
1582 def _db_set_compressed_meaning(self
, p
):
1583 # Use meaning_uri because setting meaning to something else that is not
1584 # BLOB or BYTESTRING will cause the value to be decoded from utf-8 in
1585 # datastore_types.FromPropertyPb. That would break the compressed string.
1586 p
.set_meaning_uri(_MEANING_URI_COMPRESSED
)
1587 p
.set_meaning(entity_pb
.Property
.BLOB
)
1589 def _db_set_uncompressed_meaning(self
, p
):
1591 p
.set_meaning(entity_pb
.Property
.BYTESTRING
)
1593 p
.set_meaning(entity_pb
.Property
.BLOB
)
1595 def _db_get_value(self
, v
, p
):
1596 if not v
.has_stringvalue():
1598 value
= v
.stringvalue()
1599 if p
.meaning_uri() == _MEANING_URI_COMPRESSED
:
1600 value
= _CompressedValue(value
)
1604 class TextProperty(BlobProperty
):
1605 """An unindexed Property whose value is a text string of unlimited length."""
1607 def _validate(self
, value
):
1608 if isinstance(value
, str):
1609 # Decode from UTF-8 -- if this fails, we can't write it.
1611 value
= unicode(value
, 'utf-8')
1612 except UnicodeError:
1613 raise datastore_errors
.BadValueError('Expected valid UTF-8, got %r' %
1615 elif not isinstance(value
, unicode):
1616 raise datastore_errors
.BadValueError('Expected string, got %r' %
1618 if self
._indexed
and len(value
) > _MAX_STRING_LENGTH
:
1619 raise datastore_errors
.BadValueError(
1620 'Indexed value %s must be at most %d characters' %
1621 (self
._name
, _MAX_STRING_LENGTH
))
1623 def _to_base_type(self
, value
):
1624 if isinstance(value
, unicode):
1625 return value
.encode('utf-8')
1627 def _from_base_type(self
, value
):
1628 if isinstance(value
, str):
1630 return unicode(value
, 'utf-8')
1631 except UnicodeDecodeError:
1632 # Since older versions of NDB could write non-UTF-8 TEXT
1633 # properties, we can't just reject these. But _validate() now
1634 # rejects these, so you can't write new non-UTF-8 TEXT
1636 # TODO: Eventually we should close this hole.
1639 def _db_set_uncompressed_meaning(self
, p
):
1640 if not self
._indexed
:
1641 p
.set_meaning(entity_pb
.Property
.TEXT
)
1644 class StringProperty(TextProperty
):
1645 """An indexed Property whose value is a text string of limited length."""
1650 class GeoPtProperty(Property
):
1651 """A Property whose value is a GeoPt."""
1653 def _validate(self
, value
):
1654 if not isinstance(value
, GeoPt
):
1655 raise datastore_errors
.BadValueError('Expected GeoPt, got %r' %
1658 def _db_set_value(self
, v
, p
, value
):
1659 if not isinstance(value
, GeoPt
):
1660 raise TypeError('GeoPtProperty %s can only be set to GeoPt values; '
1661 'received %r' % (self
._name
, value
))
1662 p
.set_meaning(entity_pb
.Property
.GEORSS_POINT
)
1663 pv
= v
.mutable_pointvalue()
1667 def _db_get_value(self
, v
, unused_p
):
1668 if not v
.has_pointvalue():
1671 return GeoPt(pv
.x(), pv
.y())
1674 def _unpack_user(v
):
1675 """Internal helper to unpack a User value from a protocol buffer."""
1677 email
= unicode(uv
.email().decode('utf-8'))
1678 auth_domain
= unicode(uv
.auth_domain().decode('utf-8'))
1679 obfuscated_gaiaid
= uv
.obfuscated_gaiaid().decode('utf-8')
1680 obfuscated_gaiaid
= unicode(obfuscated_gaiaid
)
1682 federated_identity
= None
1683 if uv
.has_federated_identity():
1684 federated_identity
= unicode(
1685 uv
.federated_identity().decode('utf-8'))
1687 value
= users
.User(email
=email
,
1688 _auth_domain
=auth_domain
,
1689 _user_id
=obfuscated_gaiaid
,
1690 federated_identity
=federated_identity
)
1694 class PickleProperty(BlobProperty
):
1695 """A Property whose value is any picklable Python object."""
1697 def _to_base_type(self
, value
):
1698 return pickle
.dumps(value
, pickle
.HIGHEST_PROTOCOL
)
1700 def _from_base_type(self
, value
):
1701 return pickle
.loads(value
)
1704 class JsonProperty(BlobProperty
):
1705 """A property whose value is any Json-encodable Python object."""
1709 @utils.positional(1 + BlobProperty
._positional
)
1710 def __init__(self
, name
=None, compressed
=False, json_type
=None, **kwds
):
1711 super(JsonProperty
, self
).__init
__(name
=name
, compressed
=compressed
, **kwds
)
1712 self
._json
_type
= json_type
1714 def _validate(self
, value
):
1715 if self
._json
_type
is not None and not isinstance(value
, self
._json
_type
):
1716 raise TypeError('JSON property must be a %s' % self
._json
_type
)
1718 # Use late import so the dependency is optional.
1720 def _to_base_type(self
, value
):
1724 import simplejson
as json
1725 return json
.dumps(value
)
1727 def _from_base_type(self
, value
):
1731 import simplejson
as json
1732 return json
.loads(value
)
1735 class UserProperty(Property
):
1736 """A Property whose value is a User object.
1738 Note: this exists for backwards compatibility with existing
1739 datastore schemas only; we do not recommend storing User objects
1740 directly in the datastore, but instead recommend storing the
1741 user.user_id() value.
1744 _attributes
= Property
._attributes
+ ['_auto_current_user',
1745 '_auto_current_user_add']
1747 _auto_current_user
= False
1748 _auto_current_user_add
= False
1750 @utils.positional(1 + Property
._positional
)
1751 def __init__(self
, name
=None, auto_current_user
=False,
1752 auto_current_user_add
=False, **kwds
):
1753 super(UserProperty
, self
).__init
__(name
=name
, **kwds
)
1754 # TODO: Disallow combining auto_current_user* and default?
1756 if auto_current_user
:
1757 raise ValueError('UserProperty could use auto_current_user and be '
1758 'repeated, but there would be no point.')
1759 elif auto_current_user_add
:
1760 raise ValueError('UserProperty could use auto_current_user_add and be '
1761 'repeated, but there would be no point.')
1762 self
._auto
_current
_user
= auto_current_user
1763 self
._auto
_current
_user
_add
= auto_current_user_add
1765 def _validate(self
, value
):
1766 if not isinstance(value
, users
.User
):
1767 raise datastore_errors
.BadValueError('Expected User, got %r' %
1770 def _prepare_for_put(self
, entity
):
1771 if (self
._auto
_current
_user
or
1772 (self
._auto
_current
_user
_add
and not self
._has
_value
(entity
))):
1773 value
= users
.get_current_user()
1774 if value
is not None:
1775 self
._store
_value
(entity
, value
)
1777 def _db_set_value(self
, v
, p
, value
):
1778 datastore_types
.PackUser(p
.name(), value
, v
)
1780 def _db_get_value(self
, v
, unused_p
):
1781 if not v
.has_uservalue():
1783 return _unpack_user(v
)
1786 class KeyProperty(Property
):
1787 """A Property whose value is a Key object.
1789 Optional keyword argument: kind=<kind>, to require that keys
1790 assigned to this property always have the indicated kind. May be a
1791 string or a Model subclass.
1794 _attributes
= Property
._attributes
+ ['_kind']
1798 @utils.positional(2 + Property
._positional
)
1799 def __init__(self
, *args
, **kwds
):
1800 # Support several positional signatures:
1801 # () => name=None, kind from kwds
1802 # (None) => name=None, kind from kwds
1803 # (name) => name=arg 0, kind from kwds
1804 # (kind) => name=None, kind=arg 0
1805 # (name, kind) => name=arg 0, kind=arg 1
1806 # (kind, name) => name=arg 1, kind=arg 0
1807 # The positional kind must be a Model subclass; it cannot be a string.
1811 if isinstance(arg
, basestring
):
1812 if name
is not None:
1813 raise TypeError('You can only specify one name')
1815 elif isinstance(arg
, type) and issubclass(arg
, Model
):
1816 if kind
is not None:
1817 raise TypeError('You can only specify one kind')
1819 elif arg
is not None:
1820 raise TypeError('Unexpected positional argument: %r' % (arg
,))
1823 name
= kwds
.pop('name', None)
1824 elif 'name' in kwds
:
1825 raise TypeError('You can only specify name once')
1828 kind
= kwds
.pop('kind', None)
1829 elif 'kind' in kwds
:
1830 raise TypeError('You can only specify kind once')
1832 if kind
is not None:
1833 if isinstance(kind
, type) and issubclass(kind
, Model
):
1834 kind
= kind
._get
_kind
()
1835 if isinstance(kind
, unicode):
1836 kind
= kind
.encode('utf-8')
1837 if not isinstance(kind
, str):
1838 raise TypeError('kind must be a Model class or a string')
1840 super(KeyProperty
, self
).__init
__(name
, **kwds
)
1844 def _datastore_type(self
, value
):
1845 return datastore_types
.Key(value
.urlsafe())
1847 def _validate(self
, value
):
1848 if not isinstance(value
, Key
):
1849 raise datastore_errors
.BadValueError('Expected Key, got %r' % (value
,))
1850 # Reject incomplete keys.
1852 raise datastore_errors
.BadValueError('Expected complete Key, got %r' %
1854 if self
._kind
is not None:
1855 if value
.kind() != self
._kind
:
1856 raise datastore_errors
.BadValueError(
1857 'Expected Key with kind=%r, got %r' % (self
._kind
, value
))
1859 def _db_set_value(self
, v
, unused_p
, value
):
1860 if not isinstance(value
, Key
):
1861 raise TypeError('KeyProperty %s can only be set to Key values; '
1862 'received %r' % (self
._name
, value
))
1863 # See datastore_types.PackKey
1864 ref
= value
.reference()
1865 rv
= v
.mutable_referencevalue() # A Reference
1866 rv
.set_app(ref
.app())
1867 if ref
.has_name_space():
1868 rv
.set_name_space(ref
.name_space())
1869 for elem
in ref
.path().element_list():
1870 rv
.add_pathelement().CopyFrom(elem
)
1872 def _db_get_value(self
, v
, unused_p
):
1873 if not v
.has_referencevalue():
1875 ref
= entity_pb
.Reference()
1876 rv
= v
.referencevalue()
1878 ref
.set_app(rv
.app())
1879 if rv
.has_name_space():
1880 ref
.set_name_space(rv
.name_space())
1881 path
= ref
.mutable_path()
1882 for elem
in rv
.pathelement_list():
1883 path
.add_element().CopyFrom(elem
)
1884 return Key(reference
=ref
)
1887 class BlobKeyProperty(Property
):
1888 """A Property whose value is a BlobKey object."""
1890 def _validate(self
, value
):
1891 if not isinstance(value
, datastore_types
.BlobKey
):
1892 raise datastore_errors
.BadValueError('Expected BlobKey, got %r' %
1895 def _db_set_value(self
, v
, p
, value
):
1896 if not isinstance(value
, datastore_types
.BlobKey
):
1897 raise TypeError('BlobKeyProperty %s can only be set to BlobKey values; '
1898 'received %r' % (self
._name
, value
))
1899 p
.set_meaning(entity_pb
.Property
.BLOBKEY
)
1900 v
.set_stringvalue(str(value
))
1902 def _db_get_value(self
, v
, unused_p
):
1903 if not v
.has_stringvalue():
1905 return datastore_types
.BlobKey(v
.stringvalue())
1908 # The Epoch (a zero POSIX timestamp).
1909 _EPOCH
= datetime
.datetime
.utcfromtimestamp(0)
1911 class DateTimeProperty(Property
):
1912 """A Property whose value is a datetime object.
1914 Note: Unlike Django, auto_now_add can be overridden by setting the
1915 value before writing the entity. And unlike classic db, auto_now
1916 does not supply a default value. Also unlike classic db, when the
1917 entity is written, the property values are updated to match what
1918 was written. Finally, beware that this also updates the value in
1919 the in-process cache, *and* that auto_now_add may interact weirdly
1920 with transaction retries (a retry of a property with auto_now_add
1921 set will reuse the value that was set on the first try).
1924 _attributes
= Property
._attributes
+ ['_auto_now', '_auto_now_add']
1927 _auto_now_add
= False
1929 @utils.positional(1 + Property
._positional
)
1930 def __init__(self
, name
=None, auto_now
=False, auto_now_add
=False, **kwds
):
1931 super(DateTimeProperty
, self
).__init
__(name
=name
, **kwds
)
1932 # TODO: Disallow combining auto_now* and default?
1935 raise ValueError('DateTimeProperty %s could use auto_now and be '
1936 'repeated, but there would be no point.' % self
._name
)
1938 raise ValueError('DateTimeProperty %s could use auto_now_add and be '
1939 'repeated, but there would be no point.' % self
._name
)
1940 self
._auto
_now
= auto_now
1941 self
._auto
_now
_add
= auto_now_add
1943 def _validate(self
, value
):
1944 if not isinstance(value
, datetime
.datetime
):
1945 raise datastore_errors
.BadValueError('Expected datetime, got %r' %
1949 return datetime
.datetime
.now()
1951 def _prepare_for_put(self
, entity
):
1952 if (self
._auto
_now
or
1953 (self
._auto
_now
_add
and not self
._has
_value
(entity
))):
1955 self
._store
_value
(entity
, value
)
1957 def _db_set_value(self
, v
, p
, value
):
1958 if not isinstance(value
, datetime
.datetime
):
1959 raise TypeError('DatetimeProperty %s can only be set to datetime values; '
1960 'received %r' % (self
._name
, value
))
1961 if value
.tzinfo
is not None:
1962 raise NotImplementedError('DatetimeProperty %s can only support UTC. '
1963 'Please derive a new Property to support '
1964 'alternative timezones.' % self
._name
)
1966 ival
= dt
.microseconds
+ 1000000 * (dt
.seconds
+ 24 * 3600 * dt
.days
)
1967 v
.set_int64value(ival
)
1968 p
.set_meaning(entity_pb
.Property
.GD_WHEN
)
1970 def _db_get_value(self
, v
, unused_p
):
1971 if not v
.has_int64value():
1973 ival
= v
.int64value()
1974 return _EPOCH
+ datetime
.timedelta(microseconds
=ival
)
1977 def _date_to_datetime(value
):
1978 """Convert a date to a datetime for datastore storage.
1981 value: A datetime.date object.
1984 A datetime object with time set to 0:00.
1986 if not isinstance(value
, datetime
.date
):
1987 raise TypeError('Cannot convert to datetime expected date value; '
1988 'received %s' % value
)
1989 return datetime
.datetime(value
.year
, value
.month
, value
.day
)
1992 def _time_to_datetime(value
):
1993 """Convert a time to a datetime for datastore storage.
1996 value: A datetime.time object.
1999 A datetime object with date set to 1970-01-01.
2001 if not isinstance(value
, datetime
.time
):
2002 raise TypeError('Cannot convert to datetime expected time value; '
2003 'received %s' % value
)
2004 return datetime
.datetime(1970, 1, 1,
2005 value
.hour
, value
.minute
, value
.second
,
2009 class DateProperty(DateTimeProperty
):
2010 """A Property whose value is a date object."""
2012 def _validate(self
, value
):
2013 if not isinstance(value
, datetime
.date
):
2014 raise datastore_errors
.BadValueError('Expected date, got %r' %
2017 def _to_base_type(self
, value
):
2018 assert isinstance(value
, datetime
.date
), repr(value
)
2019 return _date_to_datetime(value
)
2021 def _from_base_type(self
, value
):
2022 assert isinstance(value
, datetime
.datetime
), repr(value
)
2026 return datetime
.date
.today()
2029 class TimeProperty(DateTimeProperty
):
2030 """A Property whose value is a time object."""
2032 def _validate(self
, value
):
2033 if not isinstance(value
, datetime
.time
):
2034 raise datastore_errors
.BadValueError('Expected time, got %r' %
2037 def _to_base_type(self
, value
):
2038 assert isinstance(value
, datetime
.time
), repr(value
)
2039 return _time_to_datetime(value
)
2041 def _from_base_type(self
, value
):
2042 assert isinstance(value
, datetime
.datetime
), repr(value
)
2046 return datetime
.datetime
.now().time()
2049 class _StructuredGetForDictMixin(Property
):
2050 """Mixin class so *StructuredProperty can share _get_for_dict().
2052 The behavior here is that sub-entities are converted to dictionaries
2053 by calling to_dict() on them (also doing the right thing for
2054 repeated properties).
2056 NOTE: Even though the _validate() method in StructuredProperty and
2057 LocalStructuredProperty are identical, they cannot be moved into
2058 this shared base class. The reason is subtle: _validate() is not a
2059 regular method, but treated specially by _call_to_base_type() and
2060 _call_shallow_validation(), and the class where it occurs matters
2061 if it also defines _to_base_type().
2064 def _get_for_dict(self
, entity
):
2065 value
= self
._get
_value
(entity
)
2067 value
= [v
._to
_dict
() for v
in value
]
2068 elif value
is not None:
2069 value
= value
._to
_dict
()
2073 class StructuredProperty(_StructuredGetForDictMixin
):
2074 """A Property whose value is itself an entity.
2076 The values of the sub-entity are indexed and can be queried.
2078 See the module docstring for details.
2083 _attributes
= ['_modelclass'] + Property
._attributes
2084 _positional
= 1 + Property
._positional
# Add modelclass as positional arg.
2086 @utils.positional(1 + _positional
)
2087 def __init__(self
, modelclass
, name
=None, **kwds
):
2088 super(StructuredProperty
, self
).__init
__(name
=name
, **kwds
)
2090 if modelclass
._has
_repeated
:
2091 raise TypeError('This StructuredProperty cannot use repeated=True '
2092 'because its model class (%s) contains repeated '
2093 'properties (directly or indirectly).' %
2094 modelclass
.__name
__)
2095 self
._modelclass
= modelclass
2097 def _get_value(self
, entity
):
2098 """Override _get_value() to *not* raise UnprojectedPropertyError."""
2099 value
= self
._get
_user
_value
(entity
)
2100 if value
is None and entity
._projection
:
2101 # Invoke super _get_value() to raise the proper exception.
2102 return super(StructuredProperty
, self
)._get
_value
(entity
)
2105 def __getattr__(self
, attrname
):
2106 """Dynamically get a subproperty."""
2107 # Optimistically try to use the dict key.
2108 prop
= self
._modelclass
._properties
.get(attrname
)
2109 # We're done if we have a hit and _code_name matches.
2110 if prop
is None or prop
._code
_name
!= attrname
:
2111 # Otherwise, use linear search looking for a matching _code_name.
2112 for prop
in self
._modelclass
._properties
.values():
2113 if prop
._code
_name
== attrname
:
2116 # This is executed when we never execute the above break.
2119 raise AttributeError('Model subclass %s has no attribute %s' %
2120 (self
._modelclass
.__name
__, attrname
))
2121 prop_copy
= copy
.copy(prop
)
2122 prop_copy
._name
= self
._name
+ '.' + prop_copy
._name
2123 # Cache the outcome, so subsequent requests for the same attribute
2124 # name will get the copied property directly rather than going
2125 # through the above motions all over again.
2126 setattr(self
, attrname
, prop_copy
)
2129 def _comparison(self
, op
, value
):
2131 raise datastore_errors
.BadFilterError(
2132 'StructuredProperty filter can only use ==')
2133 if not self
._indexed
:
2134 raise datastore_errors
.BadFilterError(
2135 'Cannot query for unindexed StructuredProperty %s' % self
._name
)
2136 # Import late to avoid circular imports.
2137 from .query
import ConjunctionNode
, PostFilterNode
2138 from .query
import RepeatedStructuredPropertyPredicate
2140 from .query
import FilterNode
# Import late to avoid circular imports.
2141 return FilterNode(self
._name
, op
, value
)
2142 value
= self
._do
_validate
(value
)
2143 value
= self
._call
_to
_base
_type
(value
)
2146 # TODO: Why not just iterate over value._values?
2147 for prop
in self
._modelclass
._properties
.itervalues():
2148 vals
= prop
._get
_base
_value
_unwrapped
_as
_list
(value
)
2151 raise datastore_errors
.BadFilterError(
2152 'Cannot query for non-empty repeated property %s' % prop
._name
)
2154 assert isinstance(vals
, list) and len(vals
) == 1, repr(vals
)
2157 altprop
= getattr(self
, prop
._code
_name
)
2158 filt
= altprop
._comparison
(op
, val
)
2159 filters
.append(filt
)
2160 match_keys
.append(altprop
._name
)
2162 raise datastore_errors
.BadFilterError(
2163 'StructuredProperty filter without any values')
2164 if len(filters
) == 1:
2167 pb
= value
._to
_pb
(allow_partial
=True)
2168 pred
= RepeatedStructuredPropertyPredicate(match_keys
, pb
,
2170 filters
.append(PostFilterNode(pred
))
2171 return ConjunctionNode(*filters
)
2173 def _IN(self
, value
):
2174 if not isinstance(value
, (list, tuple, set, frozenset)):
2175 raise datastore_errors
.BadArgumentError(
2176 'Expected list, tuple or set, got %r' % (value
,))
2177 from .query
import DisjunctionNode
, FalseNode
2178 # Expand to a series of == filters.
2179 filters
= [self
._comparison
('=', val
) for val
in value
]
2181 # DisjunctionNode doesn't like an empty list of filters.
2182 # Running the query will still fail, but this matches the
2183 # behavior of IN for regular properties.
2186 return DisjunctionNode(*filters
)
2189 def _validate(self
, value
):
2190 if isinstance(value
, dict):
2191 # A dict is assumed to be the result of a _to_dict() call.
2192 return self
._modelclass
(**value
)
2193 if not isinstance(value
, self
._modelclass
):
2194 raise datastore_errors
.BadValueError('Expected %s instance, got %r' %
2195 (self
._modelclass
.__name
__, value
))
2197 def _has_value(self
, entity
, rest
=None):
2198 # rest: optional list of attribute names to check in addition.
2199 # Basically, prop._has_value(self, ent, ['x', 'y']) is similar to
2200 # (prop._has_value(ent) and
2201 # prop.x._has_value(ent.x) and
2202 # prop.x.y._has_value(ent.x.y))
2203 # assuming prop.x and prop.x.y exist.
2204 # NOTE: This is not particularly efficient if len(rest) > 1,
2205 # but that seems a rare case, so for now I don't care.
2206 ok
= super(StructuredProperty
, self
)._has
_value
(entity
)
2208 lst
= self
._get
_base
_value
_unwrapped
_as
_list
(entity
)
2210 raise RuntimeError('Failed to retrieve sub-entity of StructuredProperty'
2215 subprop
= subent
._properties
.get(rest
[0])
2219 ok
= subprop
._has
_value
(subent
, rest
[1:])
2222 def _serialize(self
, entity
, pb
, prefix
='', parent_repeated
=False,
2224 # entity -> pb; pb is an EntityProto message
2225 values
= self
._get
_base
_value
_unwrapped
_as
_list
(entity
)
2226 for value
in values
:
2227 if value
is not None:
2228 # TODO: Avoid re-sorting for repeated values.
2229 for unused_name
, prop
in sorted(value
._properties
.iteritems()):
2230 prop
._serialize
(value
, pb
, prefix
+ self
._name
+ '.',
2231 self
._repeated
or parent_repeated
,
2232 projection
=projection
)
2234 # Serialize a single None
2235 super(StructuredProperty
, self
)._serialize
(
2236 entity
, pb
, prefix
=prefix
, parent_repeated
=parent_repeated
,
2237 projection
=projection
)
2239 def _deserialize(self
, entity
, p
, depth
=1):
2240 if not self
._repeated
:
2241 subentity
= self
._retrieve
_value
(entity
)
2242 if subentity
is None:
2243 subentity
= self
._modelclass
()
2244 self
._store
_value
(entity
, _BaseValue(subentity
))
2245 cls
= self
._modelclass
2246 if isinstance(subentity
, _BaseValue
):
2247 # NOTE: It may not be a _BaseValue when we're deserializing a
2248 # repeated structured property.
2249 subentity
= subentity
.b_val
2250 if not isinstance(subentity
, cls
):
2251 raise RuntimeError('Cannot deserialize StructuredProperty %s; value '
2252 'retrieved not a %s instance %r' %
2253 (self
._name
, cls
.__name
__, subentity
))
2254 prop
= subentity
._get
_property
_for
(p
, depth
=depth
)
2256 # Special case: kill subentity after all.
2257 self
._store
_value
(entity
, None)
2259 prop
._deserialize
(subentity
, p
, depth
+ 1)
2262 # The repeated case is more complicated.
2263 # TODO: Prove we won't get here for orphans.
2265 parts
= name
.split('.')
2266 if len(parts
) <= depth
:
2267 raise RuntimeError('StructuredProperty %s expected to find properties '
2268 'separated by periods at a depth of %i; received %r' %
2269 (self
._name
, depth
, parts
))
2271 rest
= parts
[depth
+ 1:]
2272 prop
= self
._modelclass
._properties
.get(next
)
2273 prop_is_fake
= False
2275 # Synthesize a fake property. (We can't use Model._fake_property()
2276 # because we need the property before we can determine the subentity.)
2278 # TODO: Handle this case, too.
2279 logging
.warn('Skipping unknown structured subproperty (%s) '
2280 'in repeated structured property (%s of %s)',
2281 name
, self
._name
, entity
.__class
__.__name
__)
2283 # TODO: Figure out the value for indexed. Unfortunately we'd
2284 # need this passed in from _from_pb(), which would mean a
2285 # signature change for _deserialize(), which might break valid
2286 # end-user code that overrides it.
2287 compressed
= p
.meaning_uri() == _MEANING_URI_COMPRESSED
2288 prop
= GenericProperty(next
, compressed
=compressed
)
2289 prop
._code
_name
= next
2292 values
= self
._get
_base
_value
_unwrapped
_as
_list
(entity
)
2293 # Find the first subentity that doesn't have a value for this
2296 if not isinstance(sub
, self
._modelclass
):
2297 raise TypeError('sub-entities must be instances of their Model class.')
2298 if not prop
._has
_value
(sub
, rest
):
2302 # We didn't find one. Add a new one to the underlying list of
2303 # values (the list returned by
2304 # _get_base_value_unwrapped_as_list() is a copy so we
2305 # can't append to it).
2306 subentity
= self
._modelclass
()
2307 values
= self
._retrieve
_value
(entity
)
2308 values
.append(_BaseValue(subentity
))
2310 # Add the synthetic property to the subentity's _properties
2311 # dict, so that it will be correctly deserialized.
2312 # (See Model._fake_property() for comparison.)
2313 subentity
._clone
_properties
()
2314 subentity
._properties
[prop
._name
] = prop
2315 prop
._deserialize
(subentity
, p
, depth
+ 1)
2317 def _prepare_for_put(self
, entity
):
2318 values
= self
._get
_base
_value
_unwrapped
_as
_list
(entity
)
2319 for value
in values
:
2320 if value
is not None:
2321 value
._prepare
_for
_put
()
2323 def _check_property(self
, rest
=None, require_indexed
=True):
2324 """Override for Property._check_property().
2327 InvalidPropertyError if no subproperty is specified or if something
2328 is wrong with the subproperty.
2331 raise InvalidPropertyError(
2332 'Structured property %s requires a subproperty' % self
._name
)
2333 self
._modelclass
._check
_properties
([rest
], require_indexed
=require_indexed
)
2336 class LocalStructuredProperty(_StructuredGetForDictMixin
, BlobProperty
):
2337 """Substructure that is serialized to an opaque blob.
2339 This looks like StructuredProperty on the Python side, but is
2340 written like a BlobProperty in the datastore. It is not indexed
2341 and you cannot query for subproperties. On the other hand, the
2342 on-disk representation is more efficient and can be made even more
2343 efficient by passing compressed=True, which compresses the blob
2351 _attributes
= ['_modelclass'] + BlobProperty
._attributes
+ ['_keep_keys']
2352 _positional
= 1 + BlobProperty
._positional
# Add modelclass as positional.
2354 @utils.positional(1 + _positional
)
2355 def __init__(self
, modelclass
,
2356 name
=None, compressed
=False, keep_keys
=False,
2358 super(LocalStructuredProperty
, self
).__init
__(name
=name
,
2359 compressed
=compressed
,
2362 raise NotImplementedError('Cannot index LocalStructuredProperty %s.' %
2364 self
._modelclass
= modelclass
2365 self
._keep
_keys
= keep_keys
2367 def _validate(self
, value
):
2368 if isinstance(value
, dict):
2369 # A dict is assumed to be the result of a _to_dict() call.
2370 return self
._modelclass
(**value
)
2371 if not isinstance(value
, self
._modelclass
):
2372 raise datastore_errors
.BadValueError('Expected %s instance, got %r' %
2373 (self
._modelclass
.__name
__, value
))
2375 def _to_base_type(self
, value
):
2376 if isinstance(value
, self
._modelclass
):
2377 pb
= value
._to
_pb
(set_key
=self
._keep
_keys
)
2378 return pb
.SerializePartialToString()
2380 def _from_base_type(self
, value
):
2381 if not isinstance(value
, self
._modelclass
):
2382 pb
= entity_pb
.EntityProto()
2383 pb
.MergePartialFromString(value
)
2384 if not self
._keep
_keys
:
2386 return self
._modelclass
._from
_pb
(pb
)
2388 def _prepare_for_put(self
, entity
):
2389 # TODO: Using _get_user_value() here makes it impossible to
2390 # subclass this class and add a _from_base_type(). But using
2391 # _get_base_value() won't work, since that would return
2392 # the serialized (and possibly compressed) serialized blob.
2393 value
= self
._get
_user
_value
(entity
)
2394 if value
is not None:
2396 for subent
in value
:
2397 subent
._prepare
_for
_put
()
2399 value
._prepare
_for
_put
()
2401 def _db_set_uncompressed_meaning(self
, p
):
2402 p
.set_meaning(entity_pb
.Property
.ENTITY_PROTO
)
2405 class GenericProperty(Property
):
2406 """A Property whose value can be (almost) any basic type.
2408 This is mainly used for Expando and for orphans (values present in
2409 the datastore but not represented in the Model subclass) but can
2410 also be used explicitly for properties with dynamically-typed
2413 This supports compressed=True, which is only effective for str
2414 values (not for unicode), and implies indexed=False.
2419 _attributes
= Property
._attributes
+ ['_compressed']
2421 @utils.positional(1 + Property
._positional
)
2422 def __init__(self
, name
=None, compressed
=False, **kwds
):
2423 if compressed
: # Compressed implies unindexed.
2424 kwds
.setdefault('indexed', False)
2425 super(GenericProperty
, self
).__init
__(name
=name
, **kwds
)
2426 self
._compressed
= compressed
2427 if compressed
and self
._indexed
:
2428 # TODO: Allow this, but only allow == and IN comparisons?
2429 raise NotImplementedError('GenericProperty %s cannot be compressed and '
2430 'indexed at the same time.' % self
._name
)
2432 def _to_base_type(self
, value
):
2433 if self
._compressed
and isinstance(value
, str):
2434 return _CompressedValue(zlib
.compress(value
))
2436 def _from_base_type(self
, value
):
2437 if isinstance(value
, _CompressedValue
):
2438 return zlib
.decompress(value
.z_val
)
2440 def _validate(self
, value
):
2441 if (isinstance(value
, basestring
) and
2443 len(value
) > _MAX_STRING_LENGTH
):
2444 raise datastore_errors
.BadValueError(
2445 'Indexed value %s must be at most %d bytes' %
2446 (self
._name
, _MAX_STRING_LENGTH
))
2448 def _db_get_value(self
, v
, p
):
2449 # This is awkward but there seems to be no faster way to inspect
2450 # what union member is present. datastore_types.FromPropertyPb(),
2451 # the undisputed authority, has the same series of if-elif blocks.
2452 # (We don't even want to think about multiple members... :-)
2453 if v
.has_stringvalue():
2454 sval
= v
.stringvalue()
2455 meaning
= p
.meaning()
2456 if meaning
== entity_pb
.Property
.BLOBKEY
:
2457 sval
= BlobKey(sval
)
2458 elif meaning
== entity_pb
.Property
.BLOB
:
2459 if p
.meaning_uri() == _MEANING_URI_COMPRESSED
:
2460 sval
= _CompressedValue(sval
)
2461 elif meaning
== entity_pb
.Property
.ENTITY_PROTO
:
2462 # NOTE: This is only used for uncompressed LocalStructuredProperties.
2463 pb
= entity_pb
.EntityProto()
2464 pb
.MergePartialFromString(sval
)
2465 modelclass
= Expando
2466 if pb
.key().path().element_size():
2467 kind
= pb
.key().path().element(-1).type()
2468 modelclass
= Model
._kind
_map
.get(kind
, modelclass
)
2469 sval
= modelclass
._from
_pb
(pb
)
2470 elif meaning
!= entity_pb
.Property
.BYTESTRING
:
2472 sval
.decode('ascii')
2473 # If this passes, don't return unicode.
2474 except UnicodeDecodeError:
2476 sval
= unicode(sval
.decode('utf-8'))
2477 except UnicodeDecodeError:
2480 elif v
.has_int64value():
2481 ival
= v
.int64value()
2482 if p
.meaning() == entity_pb
.Property
.GD_WHEN
:
2483 return _EPOCH
+ datetime
.timedelta(microseconds
=ival
)
2485 elif v
.has_booleanvalue():
2486 # The booleanvalue field is an int32, so booleanvalue() returns
2487 # an int, hence the conversion.
2488 return bool(v
.booleanvalue())
2489 elif v
.has_doublevalue():
2490 return v
.doublevalue()
2491 elif v
.has_referencevalue():
2492 rv
= v
.referencevalue()
2494 namespace
= rv
.name_space()
2495 pairs
= [(elem
.type(), elem
.id() or elem
.name())
2496 for elem
in rv
.pathelement_list()]
2497 return Key(pairs
=pairs
, app
=app
, namespace
=namespace
)
2498 elif v
.has_pointvalue():
2500 return GeoPt(pv
.x(), pv
.y())
2501 elif v
.has_uservalue():
2502 return _unpack_user(v
)
2504 # A missing value implies null.
2507 def _db_set_value(self
, v
, p
, value
):
2508 # TODO: use a dict mapping types to functions
2509 if isinstance(value
, str):
2510 v
.set_stringvalue(value
)
2511 # TODO: Set meaning to BLOB or BYTESTRING if it's not UTF-8?
2512 # (Or TEXT if unindexed.)
2513 elif isinstance(value
, unicode):
2514 v
.set_stringvalue(value
.encode('utf8'))
2515 if not self
._indexed
:
2516 p
.set_meaning(entity_pb
.Property
.TEXT
)
2517 elif isinstance(value
, bool): # Must test before int!
2518 v
.set_booleanvalue(value
)
2519 elif isinstance(value
, (int, long)):
2520 if not (-_MAX_LONG
<= value
< _MAX_LONG
):
2521 raise TypeError('Property %s can only accept 64-bit integers; '
2522 'received %s' % value
)
2523 v
.set_int64value(value
)
2524 elif isinstance(value
, float):
2525 v
.set_doublevalue(value
)
2526 elif isinstance(value
, Key
):
2527 # See datastore_types.PackKey
2528 ref
= value
.reference()
2529 rv
= v
.mutable_referencevalue() # A Reference
2530 rv
.set_app(ref
.app())
2531 if ref
.has_name_space():
2532 rv
.set_name_space(ref
.name_space())
2533 for elem
in ref
.path().element_list():
2534 rv
.add_pathelement().CopyFrom(elem
)
2535 elif isinstance(value
, datetime
.datetime
):
2536 if value
.tzinfo
is not None:
2537 raise NotImplementedError('Property %s can only support the UTC. '
2538 'Please derive a new Property to support '
2539 'alternative timezones.' % self
._name
)
2541 ival
= dt
.microseconds
+ 1000000 * (dt
.seconds
+ 24 * 3600 * dt
.days
)
2542 v
.set_int64value(ival
)
2543 p
.set_meaning(entity_pb
.Property
.GD_WHEN
)
2544 elif isinstance(value
, GeoPt
):
2545 p
.set_meaning(entity_pb
.Property
.GEORSS_POINT
)
2546 pv
= v
.mutable_pointvalue()
2549 elif isinstance(value
, users
.User
):
2550 datastore_types
.PackUser(p
.name(), value
, v
)
2551 elif isinstance(value
, BlobKey
):
2552 v
.set_stringvalue(str(value
))
2553 p
.set_meaning(entity_pb
.Property
.BLOBKEY
)
2554 elif isinstance(value
, Model
):
2555 set_key
= value
._key
is not None
2556 pb
= value
._to
_pb
(set_key
=set_key
)
2557 value
= pb
.SerializePartialToString()
2558 v
.set_stringvalue(value
)
2559 p
.set_meaning(entity_pb
.Property
.ENTITY_PROTO
)
2560 elif isinstance(value
, _CompressedValue
):
2562 v
.set_stringvalue(value
)
2563 p
.set_meaning_uri(_MEANING_URI_COMPRESSED
)
2564 p
.set_meaning(entity_pb
.Property
.BLOB
)
2566 raise NotImplementedError('Property %s does not support %s types.' %
2567 (self
._name
, type(value
)))
2570 class ComputedProperty(GenericProperty
):
2571 """A Property whose value is determined by a user-supplied function.
2573 Computed properties cannot be set directly, but are instead generated by a
2574 function when required. They are useful to provide fields in the datastore
2575 that can be used for filtering or sorting without having to manually set the
2576 value in code - for example, sorting on the length of a BlobProperty, or
2577 using an equality filter to check if another field is not empty.
2579 ComputedProperty can be declared as a regular property, passing a function as
2580 the first argument, or it can be used as a decorator for the function that
2581 does the calculation.
2585 >>> class DatastoreFile(Model):
2586 ... name = StringProperty()
2587 ... name_lower = ComputedProperty(lambda self: self.name.lower())
2589 ... data = BlobProperty()
2591 ... @ComputedProperty
2593 ... return len(self.data)
2595 ... def _compute_hash(self):
2596 ... return hashlib.sha1(self.data).hexdigest()
2597 ... hash = ComputedProperty(_compute_hash, name='sha1')
2600 def __init__(self
, func
, name
=None, indexed
=None, repeated
=None):
2604 func: A function that takes one argument, the model instance, and returns
2607 super(ComputedProperty
, self
).__init
__(name
=name
, indexed
=indexed
,
2611 def _set_value(self
, entity
, value
):
2612 raise ComputedPropertyError("Cannot assign to a ComputedProperty")
2614 def _delete_value(self
, entity
):
2615 raise ComputedPropertyError("Cannot delete a ComputedProperty")
2617 def _get_value(self
, entity
):
2618 # About projections and computed properties: if the computed
2619 # property itself is in the projection, don't recompute it; this
2620 # prevents raising UnprojectedPropertyError if one of the
2621 # dependents is not in the projection. However, if the computed
2622 # property is not in the projection, compute it normally -- its
2623 # dependents may all be in the projection, and it may be useful to
2624 # access the computed value without having it in the projection.
2625 # In this case, if any of the dependents is not in the projection,
2626 # accessing it in the computation function will raise
2627 # UnprojectedPropertyError which will just bubble up.
2628 if entity
._projection
and self
._name
in entity
._projection
:
2629 return super(ComputedProperty
, self
)._get
_value
(entity
)
2630 value
= self
._func
(entity
)
2631 self
._store
_value
(entity
, value
)
2634 def _prepare_for_put(self
, entity
):
2635 self
._get
_value
(entity
) # For its side effects.
2638 class MetaModel(type):
2639 """Metaclass for Model.
2641 This exists to fix up the properties -- they need to know their name.
2642 This is accomplished by calling the class's _fix_properties() method.
2645 def __init__(cls
, name
, bases
, classdict
):
2646 super(MetaModel
, cls
).__init
__(name
, bases
, classdict
)
2647 cls
._fix
_up
_properties
()
2651 for _
, prop
in sorted(cls
._properties
.iteritems()):
2652 props
.append('%s=%r' % (prop
._code
_name
, prop
))
2653 return '%s<%s>' % (cls
.__name
__, ', '.join(props
))
2656 class Model(_NotEqualMixin
):
2657 """A class describing datastore entities.
2659 Model instances are usually called entities. All model classes
2660 inheriting from Model automatically have MetaModel as their
2661 metaclass, so that the properties are fixed up properly after the
2662 class once the class is defined.
2664 Because of this, you cannot use the same Property object to describe
2665 multiple properties -- you must create separate Property objects for
2666 each property. E.g. this does not work:
2668 wrong_prop = StringProperty()
2673 The kind is normally equal to the class name (exclusive of the
2674 module name or any other parent scope). To override the kind,
2675 define a class method named _get_kind(), as follows:
2677 class MyModel(Model):
2680 return 'AnotherKind'
2683 __metaclass__
= MetaModel
2685 # Class variables updated by _fix_up_properties()
2687 _has_repeated
= False
2688 _kind_map
= {} # Dict mapping {kind: Model subclass}
2690 # Defaults for instance variables.
2693 _projection
= () # Tuple of names of projected properties.
2695 # Hardcoded pseudo-property for the key.
2699 def __init__(*args
, **kwds
):
2700 """Creates a new instance of this model (a.k.a. an entity).
2702 The new entity must be written to the datastore using an explicit
2706 key: Key instance for this model. If key is used, id and parent must
2708 id: Key id for this model. If id is used, key must be None.
2709 parent: Key instance for the parent model or None for a top-level one.
2710 If parent is used, key must be None.
2711 namespace: Optional namespace.
2712 app: Optional app ID.
2713 **kwds: Keyword arguments mapping to properties of this model.
2715 Note: you cannot define a property named key; the .key attribute
2716 always refers to the entity's key. But you can define properties
2717 named id or parent. Values for the latter cannot be passed
2718 through the constructor, but can be assigned to entity attributes
2719 after the entity has been created.
2722 raise TypeError('Model constructor takes no positional arguments.')
2723 # self is passed implicitly through args so users can define a property
2726 get_arg
= self
.__get
_arg
2727 key
= get_arg(kwds
, 'key')
2728 id = get_arg(kwds
, 'id')
2729 app
= get_arg(kwds
, 'app')
2730 namespace
= get_arg(kwds
, 'namespace')
2731 parent
= get_arg(kwds
, 'parent')
2732 projection
= get_arg(kwds
, 'projection')
2734 if (id is not None or parent
is not None or
2735 app
is not None or namespace
is not None):
2736 raise datastore_errors
.BadArgumentError(
2737 'Model constructor given key= does not accept '
2738 'id=, app=, namespace=, or parent=.')
2739 self
._key
= _validate_key(key
, entity
=self
)
2740 elif (id is not None or parent
is not None or
2741 app
is not None or namespace
is not None):
2742 self
._key
= Key(self
._get
_kind
(), id,
2743 parent
=parent
, app
=app
, namespace
=namespace
)
2745 self
._set
_attributes
(kwds
)
2746 # Set the projection last, otherwise it will prevent _set_attributes().
2748 self
._set
_projection
(projection
)
2751 def __get_arg(cls
, kwds
, kwd
):
2752 """Internal helper method to parse keywords that may be property names."""
2755 return kwds
.pop(alt_kwd
)
2757 obj
= getattr(cls
, kwd
, None)
2758 if not isinstance(obj
, Property
) or isinstance(obj
, ModelKey
):
2759 return kwds
.pop(kwd
)
2762 def __getstate__(self
):
2763 return self
._to
_pb
().Encode()
2765 def __setstate__(self
, serialized_pb
):
2766 pb
= entity_pb
.EntityProto(serialized_pb
)
2768 self
.__class
__._from
_pb
(pb
, set_key
=False, ent
=self
)
2770 def _populate(self
, **kwds
):
2771 """Populate an instance from keyword arguments.
2773 Each keyword argument will be used to set a corresponding
2774 property. Keywords must refer to valid property name. This is
2775 similar to passing keyword arguments to the Model constructor,
2776 except that no provisions for key, id or parent are made.
2778 self
._set
_attributes
(kwds
)
2779 populate
= _populate
2781 def _set_attributes(self
, kwds
):
2782 """Internal helper to set attributes from keyword arguments.
2784 Expando overrides this.
2786 cls
= self
.__class
__
2787 for name
, value
in kwds
.iteritems():
2788 prop
= getattr(cls
, name
) # Raises AttributeError for unknown properties.
2789 if not isinstance(prop
, Property
):
2790 raise TypeError('Cannot set non-property %s' % name
)
2791 prop
._set
_value
(self
, value
)
2793 def _find_uninitialized(self
):
2794 """Internal helper to find uninitialized properties.
2797 A set of property names.
2800 for name
, prop
in self
._properties
.iteritems()
2801 if not prop
._is
_initialized
(self
))
2803 def _check_initialized(self
):
2804 """Internal helper to check for uninitialized properties.
2807 BadValueError if it finds any.
2809 baddies
= self
._find
_uninitialized
()
2811 raise datastore_errors
.BadValueError(
2812 'Entity has uninitialized properties: %s' % ', '.join(baddies
))
2815 """Return an unambiguous string representation of an entity."""
2817 for prop
in self
._properties
.itervalues():
2818 if prop
._has
_value
(self
):
2819 val
= prop
._retrieve
_value
(self
)
2822 elif prop
._repeated
:
2823 reprs
= [prop
._value
_to
_repr
(v
) for v
in val
]
2825 reprs
[0] = '[' + reprs
[0]
2826 reprs
[-1] = reprs
[-1] + ']'
2827 rep
= ', '.join(reprs
)
2831 rep
= prop
._value
_to
_repr
(val
)
2832 args
.append('%s=%s' % (prop
._code
_name
, rep
))
2834 if self
._key
is not None:
2835 args
.insert(0, 'key=%r' % self
._key
)
2836 if self
._projection
:
2837 args
.append('_projection=%r' % (self
._projection
,))
2838 s
= '%s(%s)' % (self
.__class
__.__name
__, ', '.join(args
))
2843 """Return the kind name for this class.
2845 This defaults to cls.__name__; users may overrid this to give a
2846 class a different on-disk name than its class name.
2851 def _class_name(cls
):
2852 """A hook for polymodel to override.
2854 For regular models and expandos this is just an alias for
2855 _get_kind(). For PolyModel subclasses, it returns the class name
2856 (as set in the 'class' attribute thereof), whereas _get_kind()
2857 returns the kind (the class name of the root class of a specific
2858 PolyModel hierarchy).
2860 return cls
._get
_kind
()
2863 def _default_filters(cls
):
2864 """Return an iterable of filters that are always to be applied.
2866 This is used by PolyModel to quietly insert a filter for the
2872 def _reset_kind_map(cls
):
2873 """Clear the kind map. Useful for testing."""
2874 # Preserve "system" kinds, like __namespace__
2876 for name
, value
in cls
._kind
_map
.iteritems():
2877 if name
.startswith('__') and name
.endswith('__'):
2879 cls
._kind
_map
.clear()
2880 cls
._kind
_map
.update(keep
)
2882 def _has_complete_key(self
):
2883 """Return whether this entity has a complete key."""
2884 return self
._key
is not None and self
._key
.id() is not None
2885 has_complete_key
= _has_complete_key
2888 """Dummy hash function.
2891 Always TypeError to emphasize that entities are mutable.
2893 raise TypeError('Model is not immutable')
2895 # TODO: Reject __lt__, __le__, __gt__, __ge__.
2897 def __eq__(self
, other
):
2898 """Compare two entities of the same class for equality."""
2899 if other
.__class
__ is not self
.__class
__:
2900 return NotImplemented
2901 if self
._key
!= other
._key
:
2902 # TODO: If one key is None and the other is an explicit
2903 # incomplete key of the simplest form, this should be OK.
2905 return self
._equivalent
(other
)
2907 def _equivalent(self
, other
):
2908 """Compare two entities of the same class, excluding keys."""
2909 if other
.__class
__ is not self
.__class
__: # TODO: What about subclasses?
2910 raise NotImplementedError('Cannot compare different model classes. '
2911 '%s is not %s' % (self
.__class
__.__name
__,
2912 other
.__class
_.__name
__))
2913 if set(self
._projection
) != set(other
._projection
):
2915 # It's all about determining inequality early.
2916 if len(self
._properties
) != len(other
._properties
):
2917 return False # Can only happen for Expandos.
2918 my_prop_names
= set(self
._properties
.iterkeys())
2919 their_prop_names
= set(other
._properties
.iterkeys())
2920 if my_prop_names
!= their_prop_names
:
2921 return False # Again, only possible for Expandos.
2922 if self
._projection
:
2923 my_prop_names
= set(self
._projection
)
2924 for name
in my_prop_names
:
2926 name
, _
= name
.split('.', 1)
2927 my_value
= self
._properties
[name
]._get
_value
(self
)
2928 their_value
= other
._properties
[name
]._get
_value
(other
)
2929 if my_value
!= their_value
:
2933 def _to_pb(self
, pb
=None, allow_partial
=False, set_key
=True):
2934 """Internal helper to turn an entity into an EntityProto protobuf."""
2935 if not allow_partial
:
2936 self
._check
_initialized
()
2938 pb
= entity_pb
.EntityProto()
2941 # TODO: Move the key stuff into ModelAdapter.entity_to_pb()?
2944 for unused_name
, prop
in sorted(self
._properties
.iteritems()):
2945 prop
._serialize
(self
, pb
, projection
=self
._projection
)
2949 def _key_to_pb(self
, pb
):
2950 """Internal helper to copy the key into a protobuf."""
2953 pairs
= [(self
._get
_kind
(), None)]
2954 ref
= key_module
._ReferenceFromPairs
(pairs
, reference
=pb
.mutable_key())
2956 ref
= key
.reference()
2957 pb
.mutable_key().CopyFrom(ref
)
2958 group
= pb
.mutable_entity_group() # Must initialize this.
2959 # To work around an SDK issue, only set the entity group if the
2960 # full key is complete. TODO: Remove the top test once fixed.
2961 if key
is not None and key
.id():
2962 elem
= ref
.path().element(0)
2963 if elem
.id() or elem
.name():
2964 group
.add_element().CopyFrom(elem
)
2967 def _from_pb(cls
, pb
, set_key
=True, ent
=None, key
=None):
2968 """Internal helper to create an entity from an EntityProto protobuf."""
2969 if not isinstance(pb
, entity_pb
.EntityProto
):
2970 raise TypeError('pb must be a EntityProto; received %r' % pb
)
2974 # A key passed in overrides a key in the pb.
2975 if key
is None and pb
.key().path().element_size():
2976 key
= Key(reference
=pb
.key())
2977 # If set_key is not set, skip a trivial incomplete key.
2978 if key
is not None and (set_key
or key
.id() or key
.parent()):
2981 indexed_properties
= pb
.property_list()
2982 unindexed_properties
= pb
.raw_property_list()
2984 for plist
in [indexed_properties
, unindexed_properties
]:
2986 if p
.meaning() == entity_pb
.Property
.INDEX_VALUE
:
2987 projection
.append(p
.name())
2988 prop
= ent
._get
_property
_for
(p
, plist
is indexed_properties
)
2989 prop
._deserialize
(ent
, p
)
2991 ent
._set
_projection
(projection
)
2994 def _set_projection(self
, projection
):
2996 for propname
in projection
:
2998 head
, tail
= propname
.split('.', 1)
2999 if head
in by_prefix
:
3000 by_prefix
[head
].append(tail
)
3002 by_prefix
[head
] = [tail
]
3003 self
._projection
= tuple(projection
)
3004 for propname
, proj
in by_prefix
.iteritems():
3005 prop
= self
._properties
.get(propname
)
3006 subval
= prop
._get
_base
_value
_unwrapped
_as
_list
(self
)
3008 assert item
is not None
3009 item
._set
_projection
(proj
)
3011 def _get_property_for(self
, p
, indexed
=True, depth
=0):
3012 """Internal helper to get the Property for a protobuf-level property."""
3014 parts
= name
.split('.')
3015 if len(parts
) <= depth
:
3016 # Apparently there's an unstructured value here.
3017 # Assume it is a None written for a missing value.
3018 # (It could also be that a schema change turned an unstructured
3019 # value into a structured one. In that case, too, it seems
3020 # better to return None than to return an unstructured value,
3021 # since the latter doesn't match the current schema.)
3024 prop
= self
._properties
.get(next
)
3026 prop
= self
._fake
_property
(p
, next
, indexed
)
3029 def _clone_properties(self
):
3030 """Internal helper to clone self._properties if necessary."""
3031 cls
= self
.__class
__
3032 if self
._properties
is cls
._properties
:
3033 self
._properties
= dict(cls
._properties
)
3035 def _fake_property(self
, p
, next
, indexed
=True):
3036 """Internal helper to create a fake Property."""
3037 self
._clone
_properties
()
3038 if p
.name() != next
and not p
.name().endswith('.' + next
):
3039 prop
= StructuredProperty(Expando
, next
)
3040 prop
._store
_value
(self
, _BaseValue(Expando()))
3042 compressed
= p
.meaning_uri() == _MEANING_URI_COMPRESSED
3043 prop
= GenericProperty(next
,
3044 repeated
=p
.multiple(),
3046 compressed
=compressed
)
3047 prop
._code
_name
= next
3048 self
._properties
[prop
._name
] = prop
3051 @utils.positional(1)
3052 def _to_dict(self
, include
=None, exclude
=None):
3053 """Return a dict containing the entity's property values.
3056 include: Optional set of property names to include, default all.
3057 exclude: Optional set of property names to skip, default none.
3058 A name contained in both include and exclude is excluded.
3060 if (include
is not None and
3061 not isinstance(include
, (list, tuple, set, frozenset))):
3062 raise TypeError('include should be a list, tuple or set')
3063 if (exclude
is not None and
3064 not isinstance(exclude
, (list, tuple, set, frozenset))):
3065 raise TypeError('exclude should be a list, tuple or set')
3067 for prop
in self
._properties
.itervalues():
3068 name
= prop
._code
_name
3069 if include
is not None and name
not in include
:
3071 if exclude
is not None and name
in exclude
:
3074 values
[name
] = prop
._get
_for
_dict
(self
)
3075 except UnprojectedPropertyError
:
3076 pass # Ignore unprojected properties rather than failing.
3081 def _fix_up_properties(cls
):
3082 """Fix up the properties by calling their _fix_up() method.
3084 Note: This is called by MetaModel, but may also be called manually
3085 after dynamically updating a model class.
3087 # Verify that _get_kind() returns an 8-bit string.
3088 kind
= cls
._get
_kind
()
3089 if not isinstance(kind
, basestring
):
3090 raise KindError('Class %s defines a _get_kind() method that returns '
3091 'a non-string (%r)' % (cls
.__name
__, kind
))
3092 if not isinstance(kind
, str):
3094 kind
= kind
.encode('ascii') # ASCII contents is okay.
3095 except UnicodeEncodeError:
3096 raise KindError('Class %s defines a _get_kind() method that returns '
3097 'a Unicode string (%r); please encode using utf-8' %
3098 (cls
.__name
__, kind
))
3099 cls
._properties
= {} # Map of {name: Property}
3100 if cls
.__module
__ == __name__
: # Skip the classes in *this* file.
3102 for name
in set(dir(cls
)):
3103 attr
= getattr(cls
, name
, None)
3104 if isinstance(attr
, ModelAttribute
) and not isinstance(attr
, ModelKey
):
3105 if name
.startswith('_'):
3106 raise TypeError('ModelAttribute %s cannot begin with an underscore '
3107 'character. _ prefixed attributes are reserved for '
3108 'temporary Model instance values.' % name
)
3109 attr
._fix
_up
(cls
, name
)
3110 if isinstance(attr
, Property
):
3111 if (attr
._repeated
or
3112 (isinstance(attr
, StructuredProperty
) and
3113 attr
._modelclass
._has
_repeated
)):
3114 cls
._has
_repeated
= True
3115 cls
._properties
[attr
._name
] = attr
3116 cls
._update
_kind
_map
()
3119 def _update_kind_map(cls
):
3120 """Update the kind map to include this class."""
3121 cls
._kind
_map
[cls
._get
_kind
()] = cls
3123 def _prepare_for_put(self
):
3124 if self
._properties
:
3125 for prop
in self
._properties
.itervalues():
3126 prop
._prepare
_for
_put
(self
)
3129 def _check_properties(cls
, property_names
, require_indexed
=True):
3130 """Internal helper to check the given properties exist and meet specified
3133 Called from query.py.
3136 property_names: List or tuple of property names -- each being a string,
3137 possibly containing dots (to address subproperties of structured
3141 InvalidPropertyError if one of the properties is invalid.
3142 AssertionError if the argument is not a list or tuple of strings.
3144 assert isinstance(property_names
, (list, tuple)), repr(property_names
)
3145 for name
in property_names
:
3146 assert isinstance(name
, basestring
), repr(name
)
3148 name
, rest
= name
.split('.', 1)
3151 prop
= cls
._properties
.get(name
)
3153 cls
._unknown
_property
(name
)
3155 prop
._check
_property
(rest
, require_indexed
=require_indexed
)
3158 def _unknown_property(cls
, name
):
3159 """Internal helper to raise an exception for an unknown property name.
3161 This is called by _check_properties(). It is overridden by
3162 Expando, where this is a no-op.
3165 InvalidPropertyError.
3167 raise InvalidPropertyError('Unknown property %s' % name
)
3169 def _validate_key(self
, key
):
3170 """Validation for _key attribute (designed to be overridden).
3173 key: Proposed Key to use for entity.
3180 # Datastore API using the default context.
3181 # These use local import since otherwise they'd be recursive imports.
3184 def _query(cls
, *args
, **kwds
):
3185 """Create a Query object for this class.
3188 distinct: Optional bool, short hand for group_by = projection.
3189 *args: Used to apply an initial filter
3190 **kwds: are passed to the Query() constructor.
3195 # Validating distinct.
3196 if 'distinct' in kwds
:
3197 if 'group_by' in kwds
:
3199 'cannot use distinct= and group_by= at the same time')
3200 projection
= kwds
.get('projection')
3203 'cannot use distinct= without projection=')
3204 if kwds
.pop('distinct'):
3205 kwds
['group_by'] = projection
3207 # TODO: Disallow non-empty args and filter=.
3208 from .query
import Query
# Import late to avoid circular imports.
3209 qry
= Query(kind
=cls
._get
_kind
(), **kwds
)
3210 qry
= qry
.filter(*cls
._default
_filters
())
3211 qry
= qry
.filter(*args
)
3216 def _gql(cls
, query_string
, *args
, **kwds
):
3217 """Run a GQL query."""
3218 from .query
import gql
# Import late to avoid circular imports.
3219 return gql('SELECT * FROM %s %s' % (cls
._class
_name
(), query_string
),
3223 def _put(self
, **ctx_options
):
3224 """Write this entity to the datastore.
3226 If the operation creates or completes a key, the entity's key
3227 attribute is set to the new, complete key.
3230 The key for the entity. This is always a complete key.
3232 return self
._put
_async
(**ctx_options
).get_result()
3235 def _put_async(self
, **ctx_options
):
3236 """Write this entity to the datastore.
3238 This is the asynchronous version of Model._put().
3240 if self
._projection
:
3241 raise datastore_errors
.BadRequestError('Cannot put a partial entity')
3242 from . import tasklets
3243 ctx
= tasklets
.get_context()
3244 self
._prepare
_for
_put
()
3245 if self
._key
is None:
3246 self
._key
= Key(self
._get
_kind
(), None)
3247 self
._pre
_put
_hook
()
3248 fut
= ctx
.put(self
, **ctx_options
)
3249 post_hook
= self
._post
_put
_hook
3250 if not self
._is
_default
_hook
(Model
._default
_post
_put
_hook
, post_hook
):
3251 fut
.add_immediate_callback(post_hook
, fut
)
3253 put_async
= _put_async
3256 def _get_or_insert(*args
, **kwds
):
3257 """Transactionally retrieves an existing entity or creates a new one.
3260 name: Key name to retrieve or create.
3263 namespace: Optional namespace.
3264 app: Optional app ID.
3265 parent: Parent entity key, if any.
3266 context_options: ContextOptions object (not keyword args!) or None.
3267 **kwds: Keyword arguments to pass to the constructor of the model class
3268 if an instance for the specified key name does not already exist. If
3269 an instance with the supplied key_name and parent already exists,
3270 these arguments will be discarded.
3273 Existing instance of Model class with the specified key name and parent
3274 or a new one that has just been created.
3276 cls
, args
= args
[0], args
[1:]
3277 return cls
._get
_or
_insert
_async
(*args
, **kwds
).get_result()
3278 get_or_insert
= _get_or_insert
3281 def _get_or_insert_async(*args
, **kwds
):
3282 """Transactionally retrieves an existing entity or creates a new one.
3284 This is the asynchronous version of Model._get_or_insert().
3286 # NOTE: The signature is really weird here because we want to support
3287 # models with properties named e.g. 'cls' or 'name'.
3288 from . import tasklets
3289 cls
, name
= args
# These must always be positional.
3290 get_arg
= cls
.__get
_arg
3291 app
= get_arg(kwds
, 'app')
3292 namespace
= get_arg(kwds
, 'namespace')
3293 parent
= get_arg(kwds
, 'parent')
3294 context_options
= get_arg(kwds
, 'context_options')
3295 # (End of super-special argument parsing.)
3296 # TODO: Test the heck out of this, in all sorts of evil scenarios.
3297 if not isinstance(name
, basestring
):
3298 raise TypeError('name must be a string; received %r' % name
)
3300 raise ValueError('name cannot be an empty string.')
3301 key
= Key(cls
, name
, app
=app
, namespace
=namespace
, parent
=parent
)
3304 def internal_tasklet():
3307 ent
= yield key
.get_async(options
=context_options
)
3309 ent
= cls(**kwds
) # TODO: Use _populate().
3311 yield ent
.put_async(options
=context_options
)
3312 raise tasklets
.Return(ent
)
3313 if in_transaction():
3314 # Run txn() in existing transaction.
3317 # Maybe avoid a transaction altogether.
3318 ent
= yield key
.get_async(options
=context_options
)
3320 # Run txn() in new transaction.
3321 ent
= yield transaction_async(txn
)
3322 raise tasklets
.Return(ent
)
3324 return internal_tasklet()
3326 get_or_insert_async
= _get_or_insert_async
3329 def _allocate_ids(cls
, size
=None, max=None, parent
=None, **ctx_options
):
3330 """Allocates a range of key IDs for this model class.
3333 size: Number of IDs to allocate. Either size or max can be specified,
3335 max: Maximum ID to allocate. Either size or max can be specified,
3337 parent: Parent key for which the IDs will be allocated.
3338 **ctx_options: Context options.
3341 A tuple with (start, end) for the allocated range, inclusive.
3343 return cls
._allocate
_ids
_async
(size
=size
, max=max, parent
=parent
,
3344 **ctx_options
).get_result()
3345 allocate_ids
= _allocate_ids
3348 def _allocate_ids_async(cls
, size
=None, max=None, parent
=None,
3350 """Allocates a range of key IDs for this model class.
3352 This is the asynchronous version of Model._allocate_ids().
3354 from . import tasklets
3355 ctx
= tasklets
.get_context()
3356 cls
._pre
_allocate
_ids
_hook
(size
, max, parent
)
3357 key
= Key(cls
._get
_kind
(), None, parent
=parent
)
3358 fut
= ctx
.allocate_ids(key
, size
=size
, max=max, **ctx_options
)
3359 post_hook
= cls
._post
_allocate
_ids
_hook
3360 if not cls
._is
_default
_hook
(Model
._default
_post
_allocate
_ids
_hook
,
3362 fut
.add_immediate_callback(post_hook
, size
, max, parent
, fut
)
3364 allocate_ids_async
= _allocate_ids_async
3367 @utils.positional(3)
3368 def _get_by_id(cls
, id, parent
=None, **ctx_options
):
3369 """Returns an instance of Model class by ID.
3371 This is really just a shorthand for Key(cls, id, ...).get().
3374 id: A string or integer key ID.
3375 parent: Optional parent key of the model to get.
3376 namespace: Optional namespace.
3377 app: Optional app ID.
3378 **ctx_options: Context options.
3381 A model instance or None if not found.
3383 return cls
._get
_by
_id
_async
(id, parent
=parent
, **ctx_options
).get_result()
3384 get_by_id
= _get_by_id
3387 @utils.positional(3)
3388 def _get_by_id_async(cls
, id, parent
=None, app
=None, namespace
=None,
3390 """Returns an instance of Model class by ID (and app, namespace).
3392 This is the asynchronous version of Model._get_by_id().
3394 key
= Key(cls
._get
_kind
(), id, parent
=parent
, app
=app
, namespace
=namespace
)
3395 return key
.get_async(**ctx_options
)
3396 get_by_id_async
= _get_by_id_async
3398 # Hooks that wrap around mutations. Most are class methods with
3399 # the notable exception of put, which is an instance method.
3401 # To use these, override them in your model class and call
3402 # super(<myclass>, cls).<hook>(*args).
3404 # Note that the pre-hooks are called before the operation is
3405 # scheduled. The post-hooks are called (by the Future) after the
3406 # operation has completed.
3408 # Do not use or touch the _default_* hooks. These exist for
3409 # internal use only.
3412 def _pre_allocate_ids_hook(cls
, size
, max, parent
):
3414 _default_pre_allocate_ids_hook
= _pre_allocate_ids_hook
3417 def _post_allocate_ids_hook(cls
, size
, max, parent
, future
):
3419 _default_post_allocate_ids_hook
= _post_allocate_ids_hook
3422 def _pre_delete_hook(cls
, key
):
3424 _default_pre_delete_hook
= _pre_delete_hook
3427 def _post_delete_hook(cls
, key
, future
):
3429 _default_post_delete_hook
= _post_delete_hook
3432 def _pre_get_hook(cls
, key
):
3434 _default_pre_get_hook
= _pre_get_hook
3437 def _post_get_hook(cls
, key
, future
):
3439 _default_post_get_hook
= _post_get_hook
3441 def _pre_put_hook(self
):
3443 _default_pre_put_hook
= _pre_put_hook
3445 def _post_put_hook(self
, future
):
3447 _default_post_put_hook
= _post_put_hook
3450 def _is_default_hook(default_hook
, hook
):
3451 """Checks whether a specific hook is in its default state.
3454 cls: A ndb.model.Model class.
3455 default_hook: Callable specified by ndb internally (do not override).
3456 hook: The hook defined by a model class using _post_*_hook.
3459 TypeError if either the default hook or the tested hook are not callable.
3461 if not hasattr(default_hook
, '__call__'):
3462 raise TypeError('Default hooks for ndb.model.Model must be callable')
3463 if not hasattr(hook
, '__call__'):
3464 raise TypeError('Hooks must be callable')
3465 return default_hook
.im_func
is hook
.im_func
3468 class Expando(Model
):
3469 """Model subclass to support dynamic Property names and types.
3471 See the module docstring for details.
3474 # Set this to False (in an Expando subclass or entity) to make
3475 # properties default to unindexed.
3476 _default_indexed
= True
3478 def _set_attributes(self
, kwds
):
3479 for name
, value
in kwds
.iteritems():
3480 setattr(self
, name
, value
)
3483 def _unknown_property(cls
, name
):
3484 # It is not an error as the property may be a dynamic property.
3487 def __getattr__(self
, name
):
3488 if name
.startswith('_'):
3489 return super(Expando
, self
).__getattr
__(name
)
3490 prop
= self
._properties
.get(name
)
3492 return super(Expando
, self
).__getattribute
__(name
)
3493 return prop
._get
_value
(self
)
3495 def __setattr__(self
, name
, value
):
3496 if (name
.startswith('_') or
3497 isinstance(getattr(self
.__class
__, name
, None), (Property
, property))):
3498 return super(Expando
, self
).__setattr
__(name
, value
)
3499 # TODO: Refactor this to share code with _fake_property().
3500 self
._clone
_properties
()
3501 if isinstance(value
, Model
):
3502 prop
= StructuredProperty(Model
, name
)
3503 elif isinstance(value
, dict):
3504 prop
= StructuredProperty(Expando
, name
)
3506 repeated
= isinstance(value
, list)
3507 indexed
= self
._default
_indexed
3508 # TODO: What if it's a list of Model instances?
3509 prop
= GenericProperty(name
, repeated
=repeated
, indexed
=indexed
)
3510 prop
._code
_name
= name
3511 self
._properties
[name
] = prop
3512 prop
._set
_value
(self
, value
)
3514 def __delattr__(self
, name
):
3515 if (name
.startswith('_') or
3516 isinstance(getattr(self
.__class
__, name
, None), (Property
, property))):
3517 return super(Expando
, self
).__delattr
__(name
)
3518 prop
= self
._properties
.get(name
)
3519 if not isinstance(prop
, Property
):
3520 raise TypeError('Model properties must be Property instances; not %r' %
3522 prop
._delete
_value
(self
)
3523 if prop
in self
.__class
__._properties
:
3524 raise RuntimeError('Property %s still in the list of properties for the '
3525 'base class.' % name
)
3526 del self
._properties
[name
]
3529 @utils.positional(1)
3530 def transaction(callback
, **ctx_options
):
3531 """Run a callback in a transaction.
3534 callback: A function or tasklet to be called.
3535 **ctx_options: Transaction options.
3537 Useful options include:
3538 retries=N: Retry up to N times (i.e. try up to N+1 times)
3539 propagation=<flag>: Determines how an existing transaction should be
3540 propagated, where <flag> can be one of the following:
3541 TransactionOptions.NESTED: Start a nested transaction (this is the
3542 default; but actual nested transactions are not yet implemented,
3543 so effectively you can only use this outside an existing transaction).
3544 TransactionOptions.MANDATORY: A transaction must already be in progress.
3545 TransactionOptions.ALLOWED: If a transaction is in progress, join it.
3546 TransactionOptions.INDEPENDENT: Always start a new parallel transaction.
3547 xg=True: On the High Replication Datastore, enable cross-group
3548 transactions, i.e. allow writing to up to 5 entity groups.
3550 WARNING: Using anything other than NESTED for the propagation flag
3551 can have strange consequences. When using ALLOWED or MANDATORY, if
3552 an exception is raised, the transaction is likely not safe to
3553 commit. When using INDEPENDENT it is not generally safe to return
3554 values read to the caller (as they were not read in the caller's
3558 Whatever callback() returns.
3561 Whatever callback() raises; datastore_errors.TransactionFailedError
3562 if the transaction failed.
3565 To pass arguments to a callback function, use a lambda, e.g.
3566 def my_callback(key, inc):
3568 transaction(lambda: my_callback(Key(...), 1))
3570 fut
= transaction_async(callback
, **ctx_options
)
3571 return fut
.get_result()
3574 @utils.positional(1)
3575 def transaction_async(callback
, **ctx_options
):
3576 """Run a callback in a transaction.
3578 This is the asynchronous version of transaction().
3580 from . import tasklets
3581 return tasklets
.get_context().transaction(callback
, **ctx_options
)
3584 def in_transaction():
3585 """Return whether a transaction is currently active."""
3586 from . import tasklets
3587 return tasklets
.get_context().in_transaction()
3590 @utils.positional(1)
3591 def transactional(_func
=None, **ctx_options
):
3592 """Decorator to make a function automatically run in a transaction.
3596 **ctx_options: Transaction options (see transaction(), but propagation
3597 default to TransactionOptions.ALLOWED).
3599 This supports two forms:
3607 @transactional(retries=1)
3611 if _func
is not None:
3612 # Form (1), vanilla.
3614 raise TypeError('@transactional() does not take positional arguments')
3615 # TODO: Avoid recursion, call outer_transactional_wrapper() directly?
3616 return transactional()(_func
)
3618 ctx_options
.setdefault('propagation',
3619 datastore_rpc
.TransactionOptions
.ALLOWED
)
3621 # Form (2), with options.
3622 def outer_transactional_wrapper(func
):
3623 @utils.wrapping(func
)
3624 def inner_transactional_wrapper(*args
, **kwds
):
3627 f
= lambda: func(*args
, **kwds
)
3628 return transaction(f
, **ctx_options
)
3629 return inner_transactional_wrapper
3630 return outer_transactional_wrapper
3633 @utils.positional(1)
3634 def non_transactional(_func
=None, allow_existing
=True):
3635 """A decorator that ensures a function is run outside a transaction.
3637 If there is an existing transaction (and allow_existing=True), the
3638 existing transaction is paused while the function is executed.
3642 allow_existing: If false, throw an exception if called from within
3643 a transaction. If true, temporarily re-establish the
3644 previous non-transactional context. Defaults to True.
3646 This supports two forms, similar to transactional().
3649 A wrapper for the decorated function that ensures it runs outside a
3652 if _func
is not None:
3653 # TODO: Avoid recursion, call outer_non_transactional_wrapper() directly?
3654 return non_transactional()(_func
)
3656 def outer_non_transactional_wrapper(func
):
3657 from . import tasklets
3658 @utils.wrapping(func
)
3659 def inner_non_transactional_wrapper(*args
, **kwds
):
3660 ctx
= tasklets
.get_context()
3661 if not ctx
.in_transaction():
3662 return func(*args
, **kwds
)
3663 if not allow_existing
:
3664 raise datastore_errors
.BadRequestError(
3665 '%s cannot be called within a transaction.' % func
.__name
__)
3667 while ctx
.in_transaction():
3668 ctx
= ctx
._parent
_context
3670 raise datastore_errors
.BadRequestError(
3671 'Context without non-transactional ancestor')
3673 tasklets
.set_context(ctx
)
3674 return func(*args
, **kwds
)
3676 tasklets
.set_context(save_ctx
)
3677 return inner_non_transactional_wrapper
3678 return outer_non_transactional_wrapper
3681 def get_multi_async(keys
, **ctx_options
):
3682 """Fetches a sequence of keys.
3685 keys: A sequence of keys.
3686 **ctx_options: Context options.
3691 return [key
.get_async(**ctx_options
) for key
in keys
]
3694 def get_multi(keys
, **ctx_options
):
3695 """Fetches a sequence of keys.
3698 keys: A sequence of keys.
3699 **ctx_options: Context options.
3702 A list whose items are either a Model instance or None if the key wasn't
3705 return [future
.get_result()
3706 for future
in get_multi_async(keys
, **ctx_options
)]
3709 def put_multi_async(entities
, **ctx_options
):
3710 """Stores a sequence of Model instances.
3713 entities: A sequence of Model instances.
3714 **ctx_options: Context options.
3719 return [entity
.put_async(**ctx_options
) for entity
in entities
]
3722 def put_multi(entities
, **ctx_options
):
3723 """Stores a sequence of Model instances.
3726 entities: A sequence of Model instances.
3727 **ctx_options: Context options.
3730 A list with the stored keys.
3732 return [future
.get_result()
3733 for future
in put_multi_async(entities
, **ctx_options
)]
3736 def delete_multi_async(keys
, **ctx_options
):
3737 """Deletes a sequence of keys.
3740 keys: A sequence of keys.
3741 **ctx_options: Context options.
3746 return [key
.delete_async(**ctx_options
) for key
in keys
]
3749 def delete_multi(keys
, **ctx_options
):
3750 """Deletes a sequence of keys.
3753 keys: A sequence of keys.
3754 **ctx_options: Context options.
3757 A list whose items are all None, one per deleted key.
3759 return [future
.get_result()
3760 for future
in delete_multi_async(keys
, **ctx_options
)]
3763 def get_indexes_async(**ctx_options
):
3764 """Get a data structure representing the configured indexes.
3767 **ctx_options: Context options.
3772 from . import tasklets
3773 ctx
= tasklets
.get_context()
3774 return ctx
.get_indexes(**ctx_options
)
3777 def get_indexes(**ctx_options
):
3778 """Get a data structure representing the configured indexes.
3781 **ctx_options: Context options.
3784 A list of Index objects.
3786 return get_indexes_async(**ctx_options
).get_result()
3789 # Update __all__ to contain all Property and Exception subclasses.
3790 for _name
, _object
in globals().items():
3791 if ((_name
.endswith('Property') and issubclass(_object
, Property
)) or
3792 (_name
.endswith('Error') and issubclass(_object
, Exception))):
3793 __all__
.append(_name
)