The dev_appserver pickled database loads into memory three different forms of each entity, two of which are encoded for quick writing to the datastore.
Queried entities are returned as copied instance data, with protocol-buffer operations.
The encoded forms are only used for writing, so why store them all the time in memory? Because every time an entity is put back into the store, all of these stored forms are updated, and the entire database is written back to file. By storing the encoded form all the time, they don't have to be generated for each write...
The following code removes the stored encoded forms, generating them on demand as descriptor properties. (file: dev_appserver/
datastore_file_stub.py)
class _StoredEntity(object):
def __init__(self, entity):
# self.protobuf = entity
self.encoded_protobuf = entity.Encode()
self.native = datastore.Entity._FromPb(entity)
@property
def protobuf(self):
# Used in _Dynamic_Get and _Dynamic_GetSchema
entity = entity_pb.EntityProto()
entity.CopyFrom(self.native.ToPb())
return entity
## @property
## def encoded_protobuf(self):
## # Used only in __WriteDatastore
## return self.protobuf.Encode()
This enabling dynamically-generated members for the two encoded forms, this reduces database size from 1.4Gb to roughly 204Mb. However, to speed up writes, the above code stores the encoded form to speed up writing, and the protocol-buffer form can be generated dynamically for each query, because usually it operates on fewer entities and is entirely in-memory (as opposed to on-disk). This results in an in-memory executable image taking up 390Mb.
This utility analyzes the pickled-form of a datastore file, optionally stripping all extra dev_appserver/datastore/stub/protobuf information from the database, meaning that all data is in primitive integers, strings, lists and dictionaries. Wherever possible, strings are converted to integers. Additionally, keys take up alot of space because they contain alot of information, including path and ancestor elements.
frauncache/examine_devappserver_datastore.py
This stripped down form, when output as a compact jsonified form, takes up just over 16Mb on disk (compared to the 23Mb for the pickled datastore file). When decoded into memory, it takes up roughly 94Mb. Of course, alot of the functionality and key qualities are left implied.
Since every datastore put results in writing everything, it's best to do everything in a transaction, which means mandating a root entity... Or, switch to
bdbdatastore, implemented as a java server.
So does bulkloading have a transactional form? Is it possible to incorporate the ancestor information into the exported entities?
----
Another project I was working on is to manipulate the intermediate bulkloader progress database files. While uploading/downloading entities, progress is stored in a database file, with an accompanying signature that identifies the particular bulkload operation so that an interrupted operation can be resumed with another invocation of the bulkload program.
If you want to rename these intermediate files, it breaks the signature because the progress database filename is stored as part of the signature, which is encoded into a database table row. This script lets you manipulate, in batch, these progress files in order to rename them.
frauncache/rename_bulkload_workfiles.py
It also lets you rename them with dynamically-interpolated values, so that the filename or path parts acquire the name of the model (kind) or particular progress database file type ('progress' or 'result')
Here's an example invocation:
$ python rename_bulkload_workfiles.py \
--new-db-name-template='.bulkload/progress/'`date +%s`'/$KIND/$DB_FILE_TYPE.sql3'
--rename-db-file bulkloader-progress-*
Labels: dev_appserver appengine datastore protocol buffer bdbdatastore pickle json bulkload
