py1010#

py1010: 1010data Insights Platform’s Python SDK#

py1010 enables making 1010data calls within Python and inspect the results in a familiar, Python-like way. To create a new session and log in, you can do something like:

>>> import py1010
>>> session=py1010.Session("http://www2.1010data.com/prod-latest/gw",
                           "USERNAME", "PASSWORD", py1010.POSSESS)

The last parameter is for the login type, which determines what happens when the application establishes a connection to the Insights Platform and another session is currently active. The login types are: py1010.KILL, py1010.NOKILL, or py1010.POSSESS.

For a group login, supply a group name after the login type parameter (or with the group= keyword). py1010 acquires a userid and logs in.

You can create queries using the XML query language.

>>> query=session.query("pub.demo.weather.hourly90",
                        '<sort cols="date,hour,id"/>')
>>> query.run()

The columns from the query are available as items or attributes of the query, and the contents of each column are available by subscripting:

>>> query['date'][0], query['temp'][0]
(datetime.date(1990, 1, 1), -12.2)

Insights platform “date” data is automatically converted to python datetime.date objects.

Most of the CSDK API is exposed through py1010; you may need to refer to the CSDK documentation for further explanation of some of the calls. Generally, in cases where the API function would return a non-zero value to indicate an error, py1010 will throw a TentenException object.

In the docstrings of this module, an asterisk “(*)” generally means that the documented method may cause network access.

Many of the parameters and return values of the methods will be “bytes” objects and not strings, because 1010data treats data as plain bytes and not as encoded strings, but see below.

Data Conversion#

When py1010 retrieves data from a 1010data database, by default it does some conversions before presenting it to you when you access it. The default conversions, which will usually be what you want, are described below, but they can be also overridden by user-specified conversion functions, to give you full control over how data is converted. For example, you may want to say:

py1010.globalConverters.aConvertFunc = py1010.bytes2string

Integer Columns#

Integer columns in 1010data can include things that aren’t strictly integers from Python’s point of view, like infinities and NaN (NA in the dataset). Also, dates are stored as integers in 1010data, and so appear as integer columns.

By default, an integer NA, infinity, or negative infinity is converted into float('nan'), float('inf'), or float('-inf'), respectively. For other values, if the format_type of the column is date, date4y, or ansidate, the value is converted into a python date object, and if the format_type is hm, the value is converted into a python time object. Otherwise it is returned as an integer. Note that these conversions may raise exceptions, if the integer would represent an invalid date, etc.

Long integer columns (K-type “j”) work the same way, except they are never converted into date or time objects.

Float Columns#

In float columns, NA, infinity, and negative infinity are presented as float('nan'), float('inf'), and float('-inf'), as you might expect. This isn’t conversion, that is how they are stored to begin with.

But float values are also used for datetime values, so if the format_type of the column is datehms24 or ansidatetime, the value is converted into a python datetime object (and the conversion may raise an exception.)

String Columns#

String values are normally returned as bytes, because 1010data stores them that way, as a sequence of bytes, which might not be valid Unicode characters or might have an unknown encoding. If the format_type of the column is json, then the value is converted into a string and then interpreted as JSON, returning a Python object of the appropriate type (usually a list or dictionary). The conversion to string or decoding of JSON may raise exceptions.

If you want the data to be returned as strings and not as bytes, you can set the conversion function of the column or the query (see below) to the supplied function py1010.bytes2string(), which attempts to decode the bytes as UTF-8 data, raising a UnicodeDecodeError on failure. Or you may write your own conversion function, perhaps to handle errors differently, etc.

Conversion Functions#

If these default conversions do not suit your purposes, or perhaps you need to handle exceptions at conversion-time, you can use your own conversion functions. There are several places the converters may be set, so they can be given a broad scope (inherited by many columns with only one setting) or a narrow one, and a narrow scope can be excepted from a broader one by setting the converter to the string "default".

The order of resolution is as follows:

Use the converter set on the Column in question, if any, unless it is the string default, in which case do the default conversions as described above.
If that is None, use the converter set on the Query to which the column belongs, unless it is the string “default”, in which case do the default conversions.
If that is None, use the converter set on the Session to which the query belongs, unless it is the string default, in which case do the default conversions.
If that is None, look up the converter in py1010.globalConverters.
If that is None (or default), do the default conversion as described above.

There are four conversion functions at each step, one for each type of column. So the Column, Query, and :class`Session` objects each have attributes called iConvertFunc, jConvertFunc, fConvertFunc, and aConvertFunc, for converting integers, long integers, floats, and strings, respectively. So does the module variable globalConverters, which acts as the top-level, final lookup.

The resulting function should be able to accept one or two arguments, viz. the value to be converted and optionally the column object (since such a formatter may want to vary its output depending on, for example, the contents of the column’s .format_dict)

py1010.flushpool(url, owner, password, group, logfile=None, mode='w')#

Flush a SAM Pool.

Invokes the “markgid” or “FlushPool” transaction on the group. This guarantees that any subsequent UIDs returned for the group haven’t been logged in before on a session and are fresh.(*)

Parameters:

url (str) – the URL of the 1010 connection
owner (str) – username of the group owner
password (str) – the password for the group
group (str) – the name of the group (pool) to be flushed
logfile (str) – Optional name of file used for logging this transaction
mode (str) – fopen-style mode for opening logfile, defaults to “w”

Raises:

TentenException – if transaction fails

py1010.resetpool(url, owner, password, group, logfile=None)#

Release a SAM Pool’s IDs.

Invokes the “resetpool” transaction on the group. This releases all the group’s UIDs.(*)

Parameters:

url – the URL of the 1010 connection
owner – username of the group owner
password – the password for the group
group – the name of the group (pool) to be reset
logfile – filename for logging, if any.

Raises:

TentenException – if transaction fails

py1010.sampoolstatus(url, owner, password, group, logfile=None)#

Check the status of a SAM pool.

Run the “samstatus” transaction on the given SAM pool and return the results.

Parameters:

url – The url of the 1010data gateway.
owner – The userid of the owner of the SAM pool.
password – The owner’s password.
group – The name of the group.
logfile – Filename to log the transaction (optional).

Returns:

a list of tuples of the form: [("groupname", "username", free, loggedin, marked)...] The groupname is the same for all the users. The “free”, “loggedin”, and “marked” values are boolean values for whether that userID is free, logged in, and/or marked.

py1010.warmpool(url, owner, password, group, queries=None, logfile=None)#

Warm a SAM Pool.

Parameters:

url – The URL of the 1010 connection
owner – Username of the group owner
password – Password for the group
group – Name of the group (SAM pool)
queries – List of BaseQuery objects
logfile – File to log transactions to (default None)

Returns:

the number of IDs warmed.

class py1010.Session(url, username, password, logintype, group=None, retry=0x40000000, wait=10, logfile=None, mode='w', oktadesc=None, authentication=None, *, authscript=None, **kwargs)#

Object representing a 1010 session.

Create with 1010 URL, username (or group ownername), password, and type of login (py1010.KILL, py1010.NOKILL, or py1010.POSSESS). If this is a SAM pool (group) login, add another parameter for the groupID. py1010 automatically acquires a UID and logs in with it.

Attempting to use a closed session, either directly or indirectly by using a class:Query that was created for it, raises a TentenClosedSessionException.

Properties (default value in parens. (w) means the value is writable, and (w-) means the value is writable but not readable)):

Variables:

messagetext – Last message text from 1010 API
rc (int) – Return code of last operation performed
timeout (tuple) – Timeout parameter tuple: (connectTimeout, timeout) (w)
reuseConnection – Reuse the same connection (True) (w-)
lenient – Treat non-conforming columns leniently (False) (w-)
ignoreSSLErrors – No exception on SSL errors (False) (w-)
APIversion – API version of this session (w)
systemVersion – System version this session is logged into
lastResponse – Last response from the session
transactions – Raw bitmask of available transactions
dbmdo – Used for calling the dbm() method with a slightly different syntax. session.dbmdo.endpoint(key1=val1, ...) is equivalent session.dbm("endpoint", key1=val1, ...), with one or two special cases (see the docstrings on the DBMDo object.)

Create or connect to a 1010 session

Create or connect to a 1010 session with the specified URL, username (or owner name for SAM Pool login), password, login type (py1010.KILL, py1010.NOKILL, or py1010.POSSESS), and group name (for SAM pool login). For SAM pool logins, a maximum number of retries and the number of seconds to wait between retries can also be specified (silently increased if below the minimum of 10s) (*)

If the password supplied is None, py1010 will prompt the user for it.

You can also create a Session object without logging in if you have the SessionID (SID) and the Encrypted Password (EPW) of an existing 1010data session. Pass in the url and username, and pass in the EPW as the password. Use py1010.POSSESS as the logintype, and add the keyword-only parameter sid=SID to pass in the session ID.

For SSO login or other external authentication, use the keyword-only parameter authentication=TAG, with TAG being a string specifying the style of authentication. See your administrator for your organization’s custom tag, if one is being used.

Parameters:

url – The 1010data gateway URL.
username – Username on 1010data or SSO, or SAM-pool owner ID.
password – Password for 1010data or SSO.
logintype – py1010.POSSESS to use an existing session or start a new one; py1010.KILL to end an existing session (if any) before starting this one; py1010.NOKILL to error if a session already exists.
group – SAM-pool group name, if applicable.
retry – Number of times to retry if no SAM-pool user-id is available (default no limit).
wait – Seconds to wait between retries for a user-id.
logfile – Filename (or python filehandle) for writing XML log.
mode – Mode for writing logfile (“a” for appending, “w” (default) for truncating.)
oktadesc – OktaLogin object for Okta logins.
authentication – (keyword-only) Authentication tag for SSO logins.
authscript – (keyword-only) External authentication script for SSO logins, if not the standard one.

addKey(self, keyobj, key='', id_='', region='', keytype=TBL_SOURCE_S3)#

Add a key to the server-side keystore.

Add a new Cloud key to the keystore under a user-chosen name.

Parameters:: keyobj – The CloudKey object to add

OR you can run with just the information for creating the CloudKey object and the function will construct it for you

Parameters:

keyobj – The user-defined name (a string) for the key.
key – The secret key.
id_ – The AWS id (AWS only).
region – The AWS region (AWS only).
keytype (SourceInfo.SrcType) – Type of key: an element of the SourceInfo.SrcType enumeration. Default SourceInfo.SrcType.S3

Raises:

TentenTransactionException if a key with this name already exists in the keystore (among other reasons.)

addTable(self, spec, sync=False)#

Add a table from data already uploaded to FTP account.

Returns:: The name of the table.
Raises:: TentenException – if transaction fails

addTableEnd(self, table)#

Close and commit a transaction addtab.

Run this function at the end of a “transactional” addtab process to indicate that you are finished uploading chunks of data and to close and save the table. Equivalent to sending a zero-byte chunk, i.e. self.addTableFeed(table, b'', False)

Parameters:: table – Name of the table to finish.

addTableFeed(self, table, data, compress=True)#

Send a chunk of data in a transactional addtab process.

In a “transactional” addtab process, you run Session.addTableSpecs() with a source spec which includes no SourceFiles. Then you use this method to send chunks of data to be added to the table, and finally call addTableEnd() to close and commit the table.

Parameters:

table – Name of the table you are uploading to.
data – The chunk of data to send, formatted as described in the SourceSpec.
compress – Whether or not to compress the data being sent (default True)

Raises:

TentenException – if transaction fails

addTableSpecs(self, sourcespec, tablespec, sync=False)#

Add a table using given specifications.

Runs the API’s addtable transaction. To initiate a “transactional” addtable, in which you upload the data in chunks, use a SourceSpec that lists no files.

Parameters:

sourcespec – A SourceSpec object giving the location and structure of the source file(s) and the columns therein.
tablespec – A TableSpec object giving the name and other metadata for the target table, as well as the columns there. If no columns are supplied here, the table column information will be copied from the source column information.

Raises:

TentenException – if transaction fails

addTableStatus(self)#

Check the status of an asynchronous upload with addTable()

Returns a tuple (status, numrecs). The status is an integer with the following meanings:

0: TENTEN_ADDTAB_IDLE

1: TENTEN_ADDTAB_FAILED

2: TENTEN_ADDTAB_INITIALIZING

3: TENTEN_ADDTAB_LOADING

4: TENTEN_ADDTAB_COMPLETED

5: TENTEN_ADDTAB_DIAGNOSED

The numrecs is the number of records (rows) uploaded so far (or in total if the upload is complete).

Raises:: TentenException – if transaction fails
Return type:: tuple(int, int)

addTableStatusList(self)#

Get a list of the status of addtable transactions.

Returns:: A list of AddtableStatus objects for all of the addtabletransactions that have been started so far in this session.
Raises:: TentenException – if transaction fails

api2(self, method, data=None, *, form='json', ignoreJSONErrors=False)#

Call the QuickApp API.

Parameters:

method – The API (QuickApp) to call.
data – If given, call with be POST with this data (json-encoded)
form – (keyword-only) Request the data in a given format. Default is “json”; other options may include “html” or “text”; the actual possibilities are determined by the server-side code.
ignoreJSONErrors – (keyword-only) If json.loads() (or decode('utf-8')) fails to decode the response, if this parameter is True, just return the response data instead of raising an exception

Returns:

Decoded json, or plain bytes from server if form is not “json”. If form is “json” and ignoreJSONErrors is True, returns the decoded json unless there is an error either in decoding the bytes to a string or in decoding the json, in which case it returns the plain bytes.

classmethod authenticateCredentials(cls, url, user, password)#

Check that the given credentials are valid.

This does not require an open session (class method) and does not create one. This returns True if the credentials are valid.(*)

autoSpec(self, data)#

Run the API’s autospecfromdata transaction on the given data.

Returns a dictionary of sourcefile information, with the source columns as dictionaries in a list under ‘columns’.

The autoSpecFile() method is probably superior in most situations.

Return type:: dict
Raises:: TentenException – if transaction fails

autoSpecFile(self, spec=None)#

Run the API’s autospec transaction on the source information provided.

Parameters:: spec – A SourceSpec object, or something that can be converted into one (a string or list of strings, a SourceFile object or list of SourceFiles, or a SourceInfo). It may also contain source columns and other information to be used as “hints” by the auto-spec.
Return type:: SourceSpec
Raises:: TentenException – if transaction fails

clearCache(self)#: Clear the 1010data cache. (*)

close(self)#

Close the session and release userID if necessary.

Closed sessions can no longer be accessed for most purposes; attempting to do so raises a TentenClosedSessionException.

createDirectory(self, pathname, title=None, users=None, uploaders=None, inherit_users=False, inherit_uploaders=False)#

Create a directory in the 1010data object tree.

Parameters:

pathname – Full name of folder to create.
title – Title of the directory (default None)
users – List of users allowed to read the directory (default None)
uploaders – List of users allowed to upload to the directory (default None)
inherit_users – If True, inherit allowed users from parent directory (ignoring any list of users given). Default False.
inherit_uploaders – If True, inherit allowed uploaders from parent directory (ignoring any list of users given). Default False.

Raises:

TentenException – if transaction fails
TentenClosedSessionException – if session is closed

dbm(self, trans, **kw)#

Call the ‘dbm’ API2 endpoint.

Parameters:

trans – The dbm transaction to invoke
kw – Key-value arguments for the given transaction

Returns:

The transaction’s response, decoded from json

Raises:

AssertionError – if the transaction fails or the result is not a dict with a return-code
KeyError – if the response does not have a value key
DBMError – if the return-code of the response indicates failure

debugAccum(self, on=True)#: Switch ACCUM (server) into (or out of) Debug mode.

delKey(self, keyobj, key='', id_='', region='', keytype=TBL_SOURCE_S3)#

Remove a key from the server-side keystore.

Remove an AWS key in the keystore, specified by name.

Parameters:: keyobj – The CloudKey object containing the name of the key you want to remove. The other information in the object is ignored.

OR you can run with just the information for creating the CloudKey object and the function will construct it for you

Parameters:

keyobj – The user-defined name (a string) for the key.
key – (optional, ignored)
id_ – (optional, ignored)
region – (optional, ignored)
keytype – (optional, ignored)

Raises:

TentenTransactionException if a key with this name does not already exist in the keystore (among other reasons.)

directory(self, dirname)#: Return a Directory object for the directory named (*)

disableLog(self)#: Disable XML logging

dropTable(self, table)#

Delete a table (*)

Raises:: TentenException – if transaction fails

enableLog(self, filename='log.log', mode='a')#

Enable XML logging

Parameters:

filename – Name of the file to log to, or a writeable file handle object (instance of IOBase)
mode – If a filename is given, fopen-type mode with which to open the file. Default a (append).

getQueryOps(self, pathname)#

Get the ops of a QuickQuery.

Returns the ops (the XML macro-code) of a saved QuickQuery in the 1010data object tree.

Parameters:: pathname – The path in the object tree to the saved QuickQuery.
Returns:: A dictionary {'base': basename, 'ops': optext}, containing the base table of the QuickQuery (if any; None if there is none) and the text of the ops saved in the query.

getRetry(self)#

Get retry parameters.

Returns a tuple: (retryMax, retryTimeMin, retryTimeRange, retryBase)

classmethod getUID(cls, url, owner, password, group)#: DEBUG ONLY: acquire a UID from a group. Leaves UID unused, claimed, and unreleasable!

logRawXML(self)#: Enable raw XML logging

memUsage(self)#: Return the memory usage of the session, in bytes (*)

mergeTablesMaterialize(self, tablelist, destpath, title=u'', sdesc=u'', ldesc=u'', link=u'', int maxdown=-1, int replace=False, users=None, char *segmentation=NULL, int sortseg=False, segbyAdvise=None, sortsegAdvise=None, int timeSeries=False, links=None, char *sort=NULL, *, inherit_users=False)#

Merge a list of tables.

Runs the CSDK tenten_MergeTablesMaterialize function. This saves the most recently-run Query. (*)

Parameters:

tablelist – List of table-names to be merged
destpath – Path in the 1010 object tree to save this table.
title – Title of the table (default “”)
sdesc – Short description (default “”)
ldesc – Long description (default “”)
link – String to be prepended to titles of linked columns (default “”)
maxdown – Maximum number of cells that can be downloaded at once. Default -1, for no limit
replace – Replace the table if it already exists (default False)
users – List of usernames with access (default None)
segmentation – Comma-separated string, a list of columns to segment on (Default None)
sortseg – Use SortSeg (True) or SegBy (False, default)
segbyAdvise – List of comma-separated column-name sequences which can be assumed to be segmented with SegBy segmentation if segmentation is performed (Default None)
sortsegAdvise – List of comma-separated column-name sequences which can be assumed to be segmented with SortSeg segmentation if segmentation is performed (Default None)
timeSeries – Generate time-series metadata (default False)
links – A list of 1010 link ops (default None)
sort – Comma-separated string, a list of columns by which to sort the contents of each segment (default None)
inherit_users – (keyword-only): If True, ignore the list of users given (if any) and set the user permissions to ‘inherit’.

Raises:

TentenException – if transaction fails

moveTable(self, oldpath, newpath)#

Move a table from one pathname to another (*)

Raises:: TentenException – if transaction fails

peekOnce(self, key, mode)#

Get the value of a server-side variable

Parameters:

key – Content of <peek> tag to send to server
mode – Value of attribute <peek mode> to send to server

Returns:

Value of the variable

Return type:

bytes

ptr(self, char *name=NULL)#

putKey(self, keyobj, key='', id_='', region='', keytype=TBL_SOURCE_S3)#

Change a key on the server-side keystore.

Alter a Cloud key in the keystore, specified by name, to contain new information.

Parameters:: keyobj – The CloudKey object with the name and updated info

OR you can run with just the information for creating the CloudKey object and the function will construct it for you

Parameters:

keyobj – The user-defined name (a string) for the key.
key – The new secret key.
id_ – The new AWS id (AWS only).
region – The new AWS region (AWS only).
keytype (SourceInfo.SrcType) – Type of key: an element of the SourceInfo.SrcType enumeration. Default SourceInfo.SrcType.S3

Raises:

TentenTransactionException if a key with this name does not already exist in the keystore (among other reasons.)

query(self, table, xml='')#

Create a query object for this session, for a given table and XML.

You can also pass an existing query object (from this or another session) to clone it in this session. The query is not run.

Return type:: Query

readKeys(self)#

Get a list of keys in the server-side keystore

Returns:: a list of CloudKey objects

relog(self, password, logintype=-1)#

Get a new Session object based on this one.

Returns a NEW session object, logging in again with the same URL and username, and the given password and logintype (default POSSESS). Other configurable properties are _not_ carried over into the new session. May be called on a closed session. (*)

Parameters:

password – Password for account; prompt for password if None.
logintype – py1010.POSSESS (default), py1010.KILL, or py1010.NOKILL.

Return type:

Session

saveTableMaterialize(self, tablepath, title=u'', sdesc=u'', ldesc=u'', link=u'', int maxdown=-1, int replace=False, int append=False, int appendable=True, int temporary=False, users=None, char *segmentation=NULL, int sortseg=False, segbyAdvise=None, sortsegAdvise=None, int timeSeries=False, links=None, char *sort=NULL, *, inherit_users=False)#

Create, replace, or append a table.

Runs the CSDK tenten_SaveTableMaterialize function. This saves the most recently-run Query. (*)

Parameters:

tablepath – Path in the 1010 object tree to save this table.
title – Title of the table (default “”)
sdesc – Short description (default “”)
ldesc – Long description (default “”)
link – String to be prepended to titles of linked columns (default “”)
maxdown – Maximum number of cells that can be downloaded at once. Default -1, for no limit
replace – Replace the table if it already exists (default False)
append – Append to the end of an existing table (default False)
appendable – Allow this table to be merged with other tables (default False)
temporary – Make this table temporary (default False)
users – List of usernames with access (default None)
segmentation – Comma-separated string, a list of columns to segment on (Default None)
sortseg – Use SortSeg (True) or SegBy (False, default)
segbyAdvise – List of comma-separated column-name sequences which can be assumed to be segmented with SegBy segmentation if segmentation is performed (Default None)
sortsegAdvise – List of comma-separated column-name sequences which can be assumed to be segmented with SortSeg segmentation if segmentation is performed (Default None)
timeSeries – Generate time-series metadata (default False)
links – A list of 1010 link ops (default None)
sort – Comma-separated string, a list of columns by which to sort the contents of each segment (default None)
inherit_users – (keyword-only): If True, ignore the list of users given (if any) and set the user permissions to ‘inherit’.

Raises:

TentenException – if transaction fails

searchTables(self, filter=None, maxresults=0)#

Search for tables with matching names. (*)

Returns:: a list of DirEntry objects.
Raises:: TentenException – if transaction fails

setRetry(self, long retryMax, long retryTimeMin, long retryTimeRange, double retryBase)#: Set retry parameters.

setUserAgent(self, agent, password)#: Set the UserAgent string. For internal use only.

setXFF(self, xff)#: Set X-Forwarded-For string

stop(self)#

Send a stop transaction

Raises:: TentenException – if transaction fails

updateObject(self, dirname)#

Refresh the session’s view of a file or a directory recursively. (*)

Raises:: TentenException – if transaction fails

uploadToFTP(self, localpath, password, remotename=None)#

Upload a file to your account’s FTP server.

Uploads a given file to the 1010data ftp account associated with the username of this session. The password must be provided.

Parameters:

localpath (str) – Path (local or absolute) to the file to be uploaded on the local system.
password (str) – Password for the FTP account. It should be the same as the password to the 1010data account (but needs to be supplied, as the password is not cached, for security reasons.)
remotename (str) – Name to give the file on the server side. Must be a plain base filename, not a path (i.e., it may not have / or \ characters in it). If not provided, defaults to the base name of the localpath.

uploadXMLTable(self, char *xmlbuf)#

Upload raw XML as a table (*)

Raises:: TentenException – if transaction fails

classmethod warmPool(cls, url, owner, password, group, queries=None, logfile=None)#: Same as py1010.warmpool().

APIversion#: The API version of this session (Configurable)

epw#: The encrypted password of this session.

ignoreSSLErrors#

Determines if you get an exception on SSL errors or ignore them

Default: False

lastResponse#: The last response from the session

messagetext#: Message text of the session.

pswd#: PSWD of a session

rc#: The return-code (integer) of the last transaction performed

realurl#: realurl: object

reuseConnection#

Determines if the same connection is reused between transactions

Default: True

sessionid#: SID of a session

sid#: The session ID of this session.

systemVersion#: The system version this session is logged into

timeout#

Connection timeout parameters, as a tuple: (connectTimeout, timeout)

Settable.

transactions#: The raw bitmask of available transactions

url#: url: object

userAgent#: The user agent for this session

username#: username: object

class py1010.Query(session, table, xml='')#

Representing 1010data Insights Platform queries and their result sets.

Created with the query() method on Session objects. It specifies the table the query operates on and the query’s XML. Before accessing the results of a query, the query’s run() method must be executed. This creates the Column objects that hold the result set and populates the query’s coldict and cols attributes.

The columns are accessible through the query by subscripting with column names; e.g., a column named “transaction” can be accessed as query['transaction']. Columns are also accessible as attributes of the query object, so query.transaction works.

If you have names that are already used for members of the class or are Python reserved words, you must use subscripting.

Accessing some methods or properties of a Query object before that query is run raises a TentenNotRunException. The relevant properties are marked with (-) in the list below.

Variables:

nrows – Number of rows in the result set (-)
win_start – Index of starting point of last-fetched window of data (-)
win_end – Index of end point of last-fetched window of data (-)
win_size – Size of window to be downloaded (configurable)
title – Title of the base table of this query (-)
sdesc – Short description of the base table of this query (-)
ldesc – Long description of the base table of this query (-)
messagetext – Last message text from the 1010data Insights Platform session
table – Name of the base table of this query
session (Session) – The Session object for this query
cols – A tuple of the columns. (-)
coldict (dict) – A dictionary of columns indexed by name (-)
lastrun – A datetime object of when this query was last run.
rows (RowIterator) – A new row iterator object each time accessed. (-)
xml – The XML of the operations for this query.
transfermode – The transfer mode (raw, (un)compressed, etc) of the query.
rawValues – If True, use the raw (internal) values for infinity and NA values in integer (and 64bit-integer) columns. If False (the default), return float('inf') or float('-inf') for infinities, and float('NaN') for NA values.

Create a query with the given session URL, table, and xml ops. Usually be called by session.query(…), you shouldn’t need to call it directly.

__len__()#: Number of columns in result

colbyname(self, name)#: Find column in this query by name.

dictslice(self, start, stop, step=1)#

Dictionary of lists, a slice of the data.

Return a dictionary of lists, the dictionary indexed on the names of the columns of the Query and the lists being slices of the columns.

>>> query.dictslice(start, stop, step)

is equivalent to

>>> {k : list(v[start:stop:step]) for k, v in query.coldict.items()}

Note that this instantiates lists, not iterators, so all the data called for is downloaded and stored in memory. Make sure the data size is small enough to fit in memory when calling this. (*)

expand(self, bufsize=20000)#: Returns the expanded XML query. Optional parameter is the size of the buffer (default 20000). Use a larger buffer if it is too small. (*)

formatted_row(self, index)#

Values formatted according to column formats.

A list of the values of all the columns at the given index, each one is formatted according to the format of that column. (*)

freshen(self)#: Call tenten_Freshen() to refresh the state of the table(s). (*)

getrow(self, n)#

Row at a given index.

A list of the values in all the columns at that index). (*)

has(self, n)#: This is computed with respect to the _last_ read made for this query, regardless of the column. It probably is not the method you were looking for.

resave(self, querypath, title='', sdesc='', ldesc='', users=None, inherit_users=False, owner='')#: Replace the Quick Query with this new version. (*)

rowasdict(self, n)#: Row at given index, presented as a dictionary. (*)

rowiterator(self, *args, **kwargs)#

Iterate over the rows of this query.

Arguments may be integers or strings, which are used to specify columns of the query (as index or colbyname). If none are specified, it’s taken to mean all the columns, in index order. The iterator returns a tuple of the values of all the specified columns, in the specified order.

This iterator is distinct and separate from ordinary access to the columns, and pulls down all the named columns together, instead of making a separate call for each column, so it will be more efficient if accessing a table row by row.

run(self)#: Execute query. Must be done before data can be accessed. (*)

save(self, querypath, title='', sdesc='', ldesc='', force=False, users=None, inherit_users=False, owner='')#

Save this query as a Quick Query (*).

Pass in force=True to write a new query or replace an existing one of the same name, if the first write fails.

saveFile(self, dest, targettype, names=False, headers=False, compression='zip')#

Save the results of a query to a file on FTP or cloud storage.

Parameters:

dest – A SourceInfo object with a single SourceFile in it, or a string, which is converted to one.
targettype – A string (or bytes) specifying the format to save, currently one of “csv”, “xlsx”, “pdf”, “tde”, “parquet”.
names – Use first row as column names? (default False)
compression – Should the file be saved compressed? Choices are “zip” (the default), “none”, or “gzip”.
headers – Use first row as column headers? (default False)

saveTableMaterialize(self, *args, **kwargs)#: Call saveTableMaterialize on the Session object; raise an Exception if this Query is not the ‘current’ one which would be saved.

saveToFTP(self, filename, sep='|', linesep='\n', namerow=False, headrow=False, compression='zip')#: Save this query to FTP. If namerow is True, first row will be taken to be column names. If headrow is True, first row (or second if namerow is also True) will be taken as column headers. (*)

cursor#: Current index

ldesc#: Short description of the base table of this query. (*)

messagetext#

Session message text.

This is the session’s message text, which is filled in by CSDK layer, and serves as some documentation of the status of the last operation performed.

modelcols#: Retrieve “model-type” columns as dictionaries (‘True’), or do not perform the complex calculation and the columns will just contain the word “MODEL” (‘False’, default)

nrows#: Total number of rows in the result

rawValues#: rawValues: object

rows#

A RowIterator that iterates over the rows of the query.

This is exactly the same as rowiterator(). Note: a new RowIterator is created for each access of this property.

sdesc#: Short description of the base table of this query (*)

title#: Title of the base table of this query (*)

transfermode#: Get the transfer mode (compressed, binary, etc) for this query.

win_end#: The index of the end point of the currently loaded window (last fetch).

win_size#

The size of window to download (configurable)

Note: you can change win_size after reading in some values, so win_start+win_size does not necessarily equal win_end.

win_start#: The index of the starting point of the currently loaded window (last fetch)

class py1010.Column(query, magic)#

Represents a tenten column. This is created by Query objects when they are run(). Column objects can be accessed as lists (though potentially very large ones) by subscripting (including ranges) or using iterators. Data is automatically fetched at need in ‘windows’ of size win_size (only one window is in memory at a given time for each row).

Variables:

name – Column name
type – Column type (‘a’, ‘i’, ‘j’, or ‘f’)
format – Format string
nrows – Number of rows in query
win_size – The win_size of the query associated with this column
colwin_start – The start of the current window of data for this column
colwin_end – The end of the current window of data for this column
ktype (int) – K type of the column
title – Column title
width (int) – Width field of format string (0 if not given)
format_dict – The format of the column as a dictionary
format_type – Type field of format string (None if not given)
dec (int) – Dec field of format string (-1 if not given)
rawValues – Handle infinity and NA values in integer columns, as described for the rawValues property of the Query class. If None (the default), use the value set for this column’s Query. Otherwise, override it for this column.

__len__()#

Length of this column.

The same as the number of rows in the query, i.e. self.query.nrows.

classmethod defaultAConv(cls, val, col, raw=False)#

Default string converter.

The function that py1010 uses to convert string columns if there is no user-specified converter.

If the column’s format indicates that it is a JSON value, attempt to render the json into a python object. If an exception is raised, re-raise it to the caller, unless the raw parameter is set to "safe", in which case just return the string value. Otherwise, return the value.

Parameters:

val – The value being converted.
col – The column containing the value.
raw – How to handle date conversion exceptions.

Returns:

A string or converted JSON value, as described above.

classmethod defaultFConv(cls, val, col, raw=False)#

Default float converter.

The function that py1010 uses to convert floating-point columns if there is no user-specified converter.

If the column’s format indicates that it is a timestamp (datehms24, ansidatetime), attempt to convert it to a datetime.datetime object. If an exception is raised, re-raise it to the caller, unless the raw parameter is set to "safe", in which case just return the integer value. Otherwise, return the value.

Parameters:

val – The value being converted.
col – The column containing the value.
raw – How to handle date conversion exceptions.

Returns:

A float or datetime value, as described above.

classmethod defaultIConv(cls, val, col, raw=False)#

Default integer converter.

The function that py1010 uses to convert integer columns if there is no user-specified converter.

If the value is K’s integer NA or integer plus or minus infinity, return the K integer value only if the column’s .rawValues attribute is True (or if it is None and the query’s .rawValues attribute is True). Otherwise, return float values for NaN or infinity.

If the column’s format indicates that it is some kind of date or time, attempt to convert it to a datetime.date or datetime.time object. If an exception is raised, re-raise it to the caller, unless the raw parameter is set to "safe", in which case just return the integer value.

Otherwise, return the value.

Parameters:

val – The value being converted.
col – The column containing the value.
raw – How to handle date conversion exceptions.

Returns:

An integer, date, time, or float value, as described above.

classmethod defaultJConv(cls, val, col, raw=False)#

Default long integer converter.

The function that py1010 uses to convert long integer columns if there is no user-specified converter.

If the value is K’s integer NA or integer plus or minus infinity, return the K integer value only if the column’s .rawValues attribute is True (or if it is None and the query’s .rawValues attribute is True). Otherwise, return float values for NaN or infinity. Otherwise, return the value.

Parameters:

val – The value being converted.
col – The column containing the value.
raw – How to handle date conversion exceptions.

Returns:

An integer or float value, as described above.

fetchrows(self, starting=None)#: Fetch a window of rows. Users should not need to call this. (*)

format_value(self, index)#

Format the value at the given index. (*)

Returns:: A string with the column’s value at the given index, formatted according to the column’s formatting, for dates and times.
Return type:: str

get(self, n, raw=False)#

The value of the column at the given row. (*)

If the row is not currently available for this column, it is fetched. The parameter raw controls translating dates, infinities, model columns, etc. If raw=False (default), values are converted to Python objects. If raw=True, no conversion is done. If raw="safe", conversion is done, unless the conversion results in an exception, in which case the raw value is returned. Note that self.rawValues is respected when raw is not True. Use raw=True to override self.rawValues for this particular value.

getaConv(self)#

Get the text “conversion function” in use for this column.

The order of resolution is:

self.aConvertFunc
self.query.aConvertFunc
self.query.session.aConvertFunc
py1010.globalConverters.aConvertFunc

If it all resolves to None, use the default.

getfConv(self)#

Get the float “conversion function” in use for this column.

The order of resolution is:

self.fConvertFunc
self.query.fConvertFunc
self.query.session.fConvertFunc
py1010.globalConverters.fConvertFunc

If it all resolves to None, use the default.

getiConv(self)#

Get the integer “conversion function” in use for this column.

The order of resolution is:

self.iConvertFunc
self.query.iConvertFunc
self.query.session.iConvertFunc
py1010.globalConverters.iConvertFunc

If it all resolves to None, use the default.

getjConv(self)#

Get the long integer “conversion function” in use for this column.

The order of resolution is:

self.jConvertFunc
self.query.jConvertFunc
self.query.session.jConvertFunc
py1010.globalConverters.jConvertFunc

If it all resolves to None, use the default.

getlocal(self, n)#

Fetch a value from the local cache only.

This is the same as column[n], except that it raises an IndexError if the value is not available in the currently-loaded window.

has(self, n)#

Is row n currently loaded?

Checks the given index within the window of values currently loaded for this column.

dec#

Decimal places specified in the format, or -1

Same as int(self.format_dict.get('dec', -1))

format#: Column format string

format_type#: Format type (self.format_dict.get('type',""))

ktype#: The K type of this column

name#: Column name

nrows#: Total number of rows

rawValues#: rawValues: object

title#: Column title (*)

type#: Column type (a, i, j, or f)

width#: Column format width (int(self.format_dict.get('width',0)))

win_size#: The win_size of the query associated with this column

class py1010.DBMDo(self, session)#

Used to implement the dbmdo attribute of Sessions.

With an open py1010 Session object session, the expression session.dbm("endpoint", key1=value1, key2=value2,...) is exactly the same as session.dbmdo.endpoint(key1=value1, key2=value2,...).

So you can say session.get_user(uid="name") and so on.

There are a very few exceptions, convenience functions added to this attribute.

session.dbmdo.list_tabs(path) returns a list of all the tables within the directory named by path, searching recursively.

session.load_tab(table_spec, sync=False) returns a tuple (path, status, msg) with the path of the table, the status code (see Session.addTableStatus()), and any message returned by the add-table command.

combine_tabs(self, source_paths, dest_path)#

Call the combine_tabs endpoint for the given tables.

Parameters:

source_paths – List of table names
dest_path – Path to which to save the combined table

list_tabs(self, path)#

List the tables within a given directory.

Searches recursively all through the given path.

Parameters:: path – Path to list.