py1010 reference#

py1010: 1010data Insights Platform’s Python SDK#

py1010 enables making 1010data calls within Python and inspect the results in a familiar, Python-like way. To create a new session and log in, you can do something like:

>>> import py1010
>>> session=py1010.Session("http://www2.1010data.com/prod-latest/gw",
...                        "USERNAME", "PASSWORD", py1010.POSSESS)

The last parameter is for the login type, which determines what happens when the application establishes a connection to the Insights Platform and another session is currently active. The login types are: py1010.KILL, py1010.NOKILL, or py1010.POSSESS.

For a group login, supply a group name after the login type parameter (or with the “group=” keyword). py1010 acquires a userid and logs in.

You can create queries using the XML query language.

>>> query=session.query("pub.demo.weather.hourly90",
...                     '<sort cols="date,hour,id"/>')
>>> query.run()

The columns from the query are available as items or attributes of the query, and the contents of each column are available by subscripting:

>>> query['date'][0], query['temp'][0]
(datetime.date(1990, 1, 1), -12.2)

Insights platform “date” data is automatically converted to python datetime.date objects.

Most of the CSDK API is exposed through py1010; you may need to refer to the CSDK documentation for further explanation of some of the calls. Generally, in cases where the API function would return a non-zero value to indicate an error, py1010 will throw a TentenException object.

In the docstrings of this module, an asterisk “(*)” generally means that the documented method may cause network access.

This module may be available for both Python2 and Python3. If you are using Python3, many of the parameters and return values of the methods will be “bytes” objects and not strings.

Data Conversion#

When py1010 retrieves data from a 1010data database, by default it does some conversions before presenting it to you when you access it. The default conversions, which will usually be what you want, are described below, but they can be also overridden by user-specified conversion functions, to give you full control over how data is converted. For example, you may want to say:

py1010.globalConverters.aConvertFunc = py1010.bytes2string

Integer Columns#

Integer columns in 1010data can include things that aren’t strictly integers from Python’s point of view, like infinities and NaN (NA in the dataset). Also, dates are stored as integers in 1010data, and so appear as integer columns.

By default, an integer NA, infinity, or negative infinity is converted into float(‘nan’), float(‘inf’), or float(‘-inf’), respectively. Otherwise, if the format_type of the column is date, date4y, or ansidate, the value is converted into a python date object, and if the format_type is “hm”, the value is converted into a python time object. Otherwise it is returned as an integer. Note that these conversions may raise exceptions, if the integer would represent an invalid date, etc.

Long integer columns (K-type “j”) work the same way, except they are never converted into date or time objects.

Float Columns#

In float columns, NA, infinity, and negative infinity are presented as float('nan'), float('inf'), and float('-inf'), as you might expect. This isn’t conversion, that is how they are stored to begin with.

But float values are also used for datetime values, so if the format_type of the column is datehms24 or ansidatetime, the value is converted into a python datetime object (and the conversion may raise an exception.)

String Columns#

String values are normally returned as bytes because 1010data stores them that way, as a sequence of bytes, which might not be valid Unicode characters or might have an unknown encoding. If the format_type of the column is “json”, then the value is converted into a string and then interpreted as JSON, returning a Python object of the appropriate type (usually a list or dictionary). The conversion to string or decoding of JSON may raise exceptions.

If you want the data to be returned as strings and not as bytes, you can set the conversion function of the column or the query (see below) to the supplied function py1010.bytes2string, which attempts to decode the bytes as UTF-8 data, raising a UnicodeDecodeError on failure. Or you may write your own conversion function, perhaps to handle errors differently, etc.

Conversion Functions#

If these default conversions do not suit your purposes, or perhaps you need to handle exceptions at conversion-time, you can use your own conversion functions. There are several places the converters may be set, so they can be given a broad scope (inherited by many columns with only one setting) or a narrow one, and a narrow scope can be excepted from a broader one by setting the converter to the string “default”.

The order of resolution is as follows:
  1. Use the converter set on the Column in question, if any, unless it is the string “default”, in which case do the default conversions as described above.

  2. If that is None, use the converter set on the Query to which the column belongs, unless it is the string “default”, in which case do the default conversions.

  3. If that is None, use the converter set on the Session to which the query belongs, unless it is the string “default”, in which case do the default conversions.

  4. If that is None, look up the converter in py1010.globalConverters.

  5. If that is None (or “default”), do the default conversion as described above.

There are four conversion functions at each step, one for each type of column. So the Column, Query, and Session objects each have attributes called iConvertFunc, jConvertFunc, fConvertFunc, and aConvertFunc, for converting integers, long integers, floats, and strings, respectively. So does the module variable globalConverters, which acts as the top-level, final lookup.

The resulting function should be able to accept one or two arguments, viz. the value to be converted and optionally the column object (since such a formatter may want to vary its output depending on, for example, the column’s .format_dict’s contents)

exception py1010.DBMError#

Exception raised when a DBM transaction fails.

exception py1010.TentenClosedSessionException#

Session was closed.

Exception for trying to access a session that has been closed.

exception py1010.TentenException#

Parent class for 1010data-related exceptions

exception py1010.TentenNotLastException#

This query was not the last one run.

Exception when something is done that would affect the “last” query, but the calling query is not that query.

exception py1010.TentenNotRunException#

Query has not yet been run.

Exception for trying to access information about a query that has not been run yet.

exception py1010.TentenPoolFullException#

Exception for failing to log in using a login pool, after the user-specified number of retries has been exhausted.

exception py1010.TentenTransactionException#

Server-side failure.

Exception raised when something fails in creating or running a transaction.

class py1010.AWSKey(name, key, id_=None, region=None, keytype=TBL_SOURCE_S3, ptr=None, *)#

Class holding AWS keys, wrapping CSDK AWSKEY structs.

Constructor for AWSKey.

Parameters:
  • name (str) – A string (or bytes): the name by which the API will refer to this key.

  • key (str) – A string/bytes containing the “secret” key.

  • id_ (str) – A string/bytes containing the AWS id (AWS only).

  • region (str) – A string/bytes containing the AWS region (AWS only).

  • keytype (SourceInfo.SrcType) – The type of key. Uses constants from SourceInfo class. Either SourceInfo.SrcType.S3 or SourceInfo.SrcType.ABS.

  • ptr – For internal use only.

id#

The AWS key id.

key#

The “secret” AWS key.

keytype#

The type of key (int)

name#

The name by which the system knows this key.

region#

The AWS region for this key.

class py1010.AddtableStatus(ptr=None)#

Status of an ongoing (asynchronous) Addtable transaction.

The status property is given in terms of the AddtableStatus.Status enumeration, viz:

AddtableStatus.Status.IDLE          (0)
AddtableStatus.Status.FAILED        (1)
AddtableStatus.Status.INITIALIZING  (2)
AddtableStatus.Status.LOADING       (3)
AddtableStatus.Status.COMPLETED     (4)
AddtableStatus.Status.DIAGNOSED     (5)

Properties:

tablepath       Name of table being uploaded to
status          Status of transaction (see above)
totalRecs       Number of records uploaded so far (or total)
expectedBytes   Number of bytes expected, if known
filteredBytes   Bytes filtered out by autocorrection
skippedBytes    Bytes skipped (not filtered)
message         Status message from server
targets         Location (a SourceInfo object) where filtered bytes
                are written
class Status(value)#

Enumeration for add-table status:

  • Status.IDLE

  • Status.FAILED

  • Status.INITIALIZING

  • Status.LOADING

  • Status.COMPLETED

  • Status.DIAGNOSED

expectedBytes#

Expected bytes to be uploaded.

filteredBytes#

Number of bytes filtered out by autocorrection.

message#

Status message returned by server.

skippedBytes#

Number of bytes skipped.

status#

Status of the transaction:

AddtableStatus.Status.IDLE          (0)
AddtableStatus.Status.FAILED        (1)
AddtableStatus.Status.INITIALIZING  (2)
AddtableStatus.Status.LOADING       (3)
AddtableStatus.Status.COMPLETED     (4)
AddtableStatus.Status.DIAGNOSED     (5)
tablepath#

Full path name of table being uploaded.

targets#

SourceInfo object describing file where filtered bytes are written.

totalBytes#

Total number of bytes uploaded (so far).

totalRecs#

Total number of records uploaded (so far).

class py1010.BaseQuery(table, xml)#

Base class for Query objects, not bound to sessions.

These objects basically contain only table and ops information, and cannot be run (as they have no session associated with them). They are used in pool-warming.

Properties:

table           Name of the base table of this query
xml             The XML of the operations for this query.
class py1010.Column(query, magic)#

Represents a 1010data column.

Created by Query objects when they are run. Column objects can be accessed as if they were lists (though potentially very large ones) by subscripting (including ranges) or using iterators. Data is automatically fetched at need in ‘windows’ of size win_size (only one window is in memory at a given time for each row).

Properties:

name

Column name

type

Column type (‘a’, ‘i’, ‘j’, or ‘f’)

format

Format string

nrows

Number of rows in query

win_size

The win_size of the query associated with this column

colwin_start

The start of the current window of data for this column

colwin_end

The end of the current window of data for this column

ktype

K type of the column (integer)

title

Column title

width

Width field of format string (integer; 0 if not given)

format_dict

The format of the column as a dictionary

format_type

Type field of format string (None if not given)

dec

Dec field of format string (integer; -1 if not given)

rawValues

Handle infinity and NA values in integer columns, as described for the rawValues property of the Query class. If None (the default), use the value set for this column’s Query. Otherwise, override it for this column. See also Data Conversion.

defaultAConv(type cls, val, col, raw=False)#

Default string converter.

The function that py1010 uses to convert string columns if there is no user-specified converter. (Class method)

If the Column’s format indicates that it is a JSON value, attempt to render the json into a python object. If an exception is raised, re-raise it to the caller, unless the raw parameter is set to "safe", in which case just return the string value. Otherwise, return the value.

Parameters:
  • val – The value being converted.

  • col (Column) – The column containing the value.

  • raw – How to handle date conversion exceptions.

Returns:

A string or converted JSON value, as described above.

defaultFConv(type cls, val, col, raw=False)#

Default float converter.

The function that py1010 uses to convert floating-point columns if there is no user-specified converter. (Class method)

If the column’s format indicates that it is a timestamp (datehms24, ansidatetime), attempt to convert it to a datetime.datetime object. If an exception is raised, re-raise it to the caller, unless the raw parameter is set to "safe", in which case just return the integer value. Otherwise, return the value.

Parameters:
  • val – The value being converted.

  • col (Column) – The column containing the value.

  • raw – How to handle date conversion exceptions.

Returns:

A float or datetime value, as described above.

defaultIConv(type cls, val, col, raw=False)#

Default integer converter.

The function that py1010 uses to convert integer columns if there is no user-specified converter. (Class method)

  1. If the value is K’s integer NA or integer plus or minus infinity, return the K integer value only if the Column’s .rawValues attribute is True (or if it is None and the Query’s .rawValues attribute is True). Otherwise, return float values for NaN or infinity.

  2. If the column’s format indicates that it is some kind of date or time, attempt to convert it to a datetime.date or datetime.time object. If an exception is raised, re-raise it to the caller, unless the raw parameter is set to "safe", in which case just return the integer value.

  3. Otherwise, return the value.

Parameters:
  • val – The value being converted.

  • col (Column) – The column containing the value.

  • raw – How to handle date conversion exceptions.

Returns:

An integer, date, time, or float value, as described above.

defaultJConv(type cls, val, col, raw=False)#

Default long integer converter.

The function that py1010 uses to convert long integer columns if there is no user-specified converter. (Class method)

If the value is K’s integer NA or integer plus or minus infinity, return the K integer value only if the Column’s .rawValues attribute is True (or if it is None and the Query’s .rawValues attribute is True). Otherwise, return float values for NaN or infinity. Otherwise, return the value.

Parameters:
  • val – The value being converted.

  • col (Column) – The column containing the value.

  • raw – How to handle date conversion exceptions.

Returns:

An integer or float value, as described above.

fetchrows(self, starting=None)#

Fetch a window of rows. Users should not need to call this. (*)

format_value(self, index)#

Format the value at the given index. (*)

Returns:

The value of the column at the given index as a string, formatted according to the column’s formatting, for dates and times.

Return type:

str

get(self, n, raw=False)#

The value of the column at the given row. (*)

If the row is not currently available for this column, it is fetched. The parameter “raw” controls translating dates, infinities, model columns, etc. If raw=False (default), values are converted to Python objects. If raw=True, no conversion is done. If raw="safe", conversion is done, unless the conversion results in an exception, in which case the raw value is returned. Note that self.rawValues is respected when raw is not True. Use raw=True to override self.rawValues for this particular value.

Raises:

IndexError if n is greater than the length of the column.

getaConv(self)#

Get the text “conversion function” in use for this column.

The order of resolution is:
  1. self.aConvertFunc

  2. self.query.aConvertFunc

  3. self.query.session.aConvertFunc

  4. py1010.globalConverters.aConvertFunc

If it all resolves to None, use the default.

getfConv(self)#

Get the float “conversion function” in use for this column.

The order of resolution is:
  1. self.fConvertFunc

  2. self.query.fConvertFunc

  3. self.query.session.fConvertFunc

  4. py1010.globalConverters.fConvertFunc

If it all resolves to None, use the default.

getiConv(self)#

Get the integer “conversion function” in use for this column.

The order of resolution is::
  1. self.iConvertFunc

  2. self.query.iConvertFunc

  3. self.query.session.iConvertFunc

  4. py1010.globalConverters.iConvertFunc

If it all resolves to None, use the default.

getjConv(self)#

Get the long integer “conversion function” in use for this column.

The order of resolution is:
  1. self.jConvertFunc

  2. self.query.jConvertFunc

  3. self.query.session.jConvertFunc

  4. py1010.globalConverters.jConvertFunc

If it all resolves to None, use the default.

getlocal(self, n)#

Fetch a value from the local cache only.

This is the same as column[n], except that it raises an IndexError if the value is not available in the currently-loaded window.

Raises:

IndexError if index n is not available in the currently-loaded window.

has(self, n)#

Is row n currently loaded?

Checks if the given index is within the window of values currently loaded for this column.

Return type:

bool

dec#

Decimal places specified in the format, or -1 if not set.

int(self.format_dict.get('dec', -1))

format#

Column format string

format_type#

Format type

self.format_dict.get('type',"")

ktype#

The K type of this column

name#

Column name

nrows#

Total number of rows

rawValues#

rawValues: object

title#

Column title (*)

type#

Column type (a, i, j, or f)

width#

Column format width

int(self.format_dict.get('width',0))

win_size#

The win_size of the query associated with this column

class py1010.ColumnInfo(name, type_, title=None, desc=None, format=None, int index=Index.NOINDEX, int fix=Fix.NOFIX, ptr=None, *)#

Class for holding metadata about a column being uploaded into.

class Fix(value)#

Enumeration for fix data:

  • Fix.FIX

  • Fix.NOFIX

class Index(value)#

Enumeration for indexing:

  • Index.INDEX

  • Index.NOINDEX

fromSourceCol(type cls, scol)#

Translate a SourceColumnInfo into a ColumnInfo by copying over some key fields.

init(self, name, type_, title=None, desc=None, format=None, int index=Index.NOINDEX, int fix=Fix.NOFIX)#
desc#

Description of the column

format#

Format of the column.

name#

Name of the column

title#

Title of the column, displayed when the table is viewed.

type#

Type of the column (“integer”, “yyyymmdd”, etc.)

class py1010.DBMDo(self, session)#

Used to implement the dbmdo attribute of Sessions.

With an open py1010 Session object session, the expression session.dbm("endpoint", key1=value1, key2=value2,...) is exactly the same as session.dbmdo.endpoint(key1=value1, key2=value2,...).

So you can say session.get_user(uid="name") and so on.

There are a very few exceptions, convenience functions added to this attribute.

session.dbmdo.list_tabs(path) returns a list of all the tables within the directory named by path, searching recursively.

session.load_tab(table_spec, sync=False) returns a tuple (path, status, msg) with the path of the table, the status code (see Session.addTableStatus), and any message returned by the add-table command.

combine_tabs(self, source_paths, dest_path)#

Call the combine_tabs endpoint for the given tables.

Parameters:
  • source_paths (list(str)) – List of table names

  • dest_path (str) – Path to which to save the combined table

list_tabs(self, path)#

List the tables within a given directory.

Searches recursively all through the given path.

Parameters:

path (str) – Path to list.

class py1010.DataHierarchy#

Base class for objects that convert.

See Data Conversion for details. This is just a base class providing the instance variables for the conversion functions.

aConvertFunc#

aConvertFunc: object

fConvertFunc#

fConvertFunc: object

iConvertFunc#

iConvertFunc: object

jConvertFunc#

jConvertFunc: object

class py1010.DirEntry(session, magic)#

An entry in the 1010data object tree.

A wrapper for the directory entry C structure that should only be created internally.

Properties:

type             0 for directory, 1 for table
id               id of this object
name             file/directory name
title            title of object
sdesc            short description
ldesc            long description
owner            owner of this object
update           last update timestamp
secure           if 1, object only accessible via SSL connection
asdict(self)#

The information on this object as a dictionary.

metaInfo(self)#

Get the MetaData for this directory entry (*)

Return type:

MetaData

Raises:

TentenException if the transaction fails.

setMetaInfo(self, MetaData meta, flags=-1)#

Set the MetaData on this directory entry (*)

Parameters:
  • meta (MetaData) – MetaData object to set.

  • flags (int) – Bitmask for table elements to set; defaults to -1 (all).

Raises:

TentenException if the transaction fails.

id#

ID of this object

ldesc#

Long description

name#

File or directory name

owner#

The object’s owner

sdesc#

Short description

title#

Title of this object

type#

Type of entry: 0 for directory and 1 for table(?)

update#

Update timestamp, as a datetime object

class py1010.Directory(Session session, magic)#

Represents a directory in the 1010 file tree.

You should not create these directly; use the fromPath() static method instead, or call Session.directory() on a Session object.

children(self)#

Iterate over the children of this directory (*)

Return type:

Iterator[DirEntry]

Raises:

TentenException if the transaction fails.

static createDirectory(session, pathname, title=None, users=None, uploaders=None, inherit_users=False, inherit_uploaders=False)#

Create a directory from a pathname. (*)

Static method.

Parameters:
  • session (Session) – Logged-in Session object.

  • pathname (str) – Name of directory to create

  • title (str) – Title of the directory to create

  • users (list(str)) – Users that have access to the directory (optional)

  • uploaders (list(str)) – Users that can upload to the directory (optional)

  • inherit_users (bool) – Boolean indicating whether users should be inherited. Ignored if users is non-empty.

  • inherit_uploaders (bool) – Boolean indicating whether uploaders should be inherited. Ignored if uploaders is non-empty.

Raises:

TentenException if the transaction fails.

static fromPath(session, pathname)#

Find a directory from its pathname. (*)

Static method.

Parameters:
  • session (Session) – Logged-in Session object.

  • pathname (str) – Name of the directory to fetch.

Return type:

Directory

Raises:

TentenException if the transaction fails.

parents(self)#

Iterate over the parents of this directory (*)

Return type:

Iterator[DirEntry]

Raises:

TentenException if the transaction fails.

static removeDirectory(Session session, pathname)#

Remove a directory given the pathname. (*)

Static method.

Parameters:
  • session (Session) – Logged-in Session object.

  • pathname (str) – Pathname of directory to remove.

Raises:

TentenException if the transaction fails.

class py1010.Holder(magic, size, ptr=None, *)#

Parent class for classes that will wrap a C struct.

Contains a StructWrapper which does the actual wrapping.

Superclass constructor for Holder.

Parameters:
  • magic – The magic number you have to know to use this.

  • size (int) – The size of the structure to be wrapped.

  • ptr – A PyCapsule holding a pointer to a C struct to be wrapped. If None (not supplied), allocate a new one.

PTR#

A PyCapsule holding the struct’s pointer. Internal use only.

class py1010.MetaData(session, magic)#

Contains metadata for objects in the 1010 directory system.

These are returned by the DirEntry.metaInfo() method on DirEntry objects. Various attributes of the metadata object are configurable, and they can be altered and used in DirEntry.setMetaInfo(), which changes the metadata on the file entry. Note: changing the values on the metadata object does not alter the actual values on the server, until setMetaInfo() is called.

This class is not meant to be used directly; either use the DirEntry.metaInfo() method on a DirEntry object or use the fromPath() class method.

Properties:

id               Object's id in 1010 system
path             Path-name in 1010 filetree
title            Object's title
sdesc            Short description
ldesc            Long description
type             Item type ("REAL", "DIR", etc)
secure
own
owner            Item's owner
update           Timestamp of last update
favorite
users            Users authorized to access this object
display
report
chart
link             Link description
rows             Number of rows (if a table)
bytes            Size of item (in bytes)
segs             Number of segments
tstat
access
maxdown          Maximum number of rows permitted per download
upload
uploaders
numchild
segby
keys
asdict(self)#

The metadata in dictionary form.

fromPath(type cls, Session session, path)#

Fetch the metadata for the named table or directory. (*)

Class method.

Parameters:
  • session (Session) – Logged-in Session object.

  • path (str) – Pathname of the directory or table whose data to fetch.

Return type:

MetaData

Raises:

TentenException if the transaction fails.

bytes#

size of the item (in bytes)

id#

Object’s id in the 1010 system.

ldesc#

Long description.

link description.

maxdown#

maximum number of rows permitted per download

owner#

Item’s owner

path#

Path (directories and filename)

rows#

number of rows (for a table)

sdesc#

Short description

segs#

number of segments

title#

Item’s title.

type#

Item type.

update#

Timestamp of item’s last update.

users#

Users are authorized to access this object.

class py1010.OktaLogin(email, password, gusurl='http://gus.1010data.com')#

Used by Session in order to login using okta credentials.

This object can be passed to Session in the constructor or a new Session could be created by calling getSession. This object fetches the okta token and the available environments for the user account. A new okta token and environment list are fetched from GUS, and all the environment information is available under the environments property. Using a different environment other the default one is done by changing the chosenEnvId property. Changing the environment version is done by changing the chosenVersion property.

Create OktaLogin object

Parameters:
  • email (str) – The user email

  • password (str) – User pasword

  • gusurl (str) – The GUS url to fetch the okta token from

getSession(self, logintype)#

Get new Session initializied with this OktaLogin object.

Parameters:

logintype – KILL, NOKILL, or POSSESS

sessionLogin(self, logintype, Session session)#
class py1010.Query(session, table, xml='')#

Representing 1010data Insights Platform queries and their result sets.

Created with the Session.query() method on Session objects. It specifies the table the query operates on and the query’s XML. Before accessing the results of a query, the query’s run() method must be executed. This creates the Column objects that hold the result set and populates the query’s coldict and cols attributes.

The columns are accessible through the query by subscripting with column names. A column named “transaction” can be accessed as query['transaction']. Columns are also accessible as attributes of the query object, so query.transaction works.

If you have names that are already used for members of the class or are Python reserved words, you must use subscripting.

Accessing some methods or properties of a Query object before that query is run raises a TentenNotRunException. The relevant properties are marked with (-) in the list below.

Properties:
nrows

Number of rows in the result set (-)

win_start

Index of starting point of last-fetched window of data (-)

win_end

Index of end point of last-fetched window of data (-)

win_size

Size of window to be downloaded (configurable)

title

Title of the base table of this query (-)

sdesc

Short description of the base table of this query (-)

ldesc

Long description of the base table of this query (-)

messagetext

Last message text from the 1010data Insights Platform session

table

Name of the base table of this query

session

The Session object for this query

cols

A list of the columns. (-)

coldict

A dictionary of columns indexed by name (-)

lastrun

A datetime object of when this query was last run.

rows

A new row iterator object each time accessed. (-)

xml

The XML of the operations for this query.

transfermode

The transfer mode (raw, (un)compressed, etc) of the query.

rawValues

If True, use the raw (internal) values for infinity and NA values in integer (and 64bit-integer) columns. If False (the default), return float('inf') or float('-inf') for infinities, and float('NaN') for NA values. See Data Conversion for finer control over this.

Create a query with the given session URL, table, and xml ops.

Query objects are usually created by session.query(...), you shouldn’t need to call the constructor directly.

colbyname(self, name)#

Find column in this query by name.

Same as self.coldict[name].

dictslice(self, start, stop, step=1)#

Dictionary of lists, a slice of the data. (*)

Return a dictionary of lists, the dictionary indexed on the names of the columns of the Query and the lists being slices of the columns.

>>> query.dictslice(start, stop, step)

is equivalent to

>>> {k : list(v[start:stop:step]) for k, v in query.coldict.items()}

Note that this instantiates lists, not iterators, so all the data called for is downloaded and stored in memory. Make sure the data size is small enough to fit in memory when calling this.

expand(self, bufsize=20000)#

Returns the expanded XML query. (*)

Optional parameter is the size of the buffer (default 20000). Use a larger buffer if it is too small.

formatted_row(self, index)#

Values formatted according to column formats. (*)

A list of the values of all the columns at the given index, each one is formatted according to the format of that column.

freshen(self)#

Call tenten_Freshen() to refresh the state of the table(s). (*)

Raises:

TentenException if the transaction fails.

getrow(self, n)#

Row at a given index. (*)

A list of the values in all the columns at that index).

has(self, n)#

Was the given index read in recently?

This is computed with respect to the last read made for this query, regardless of the column. It probably is not the method you were looking for.

resave(self, querypath, title='', sdesc='', ldesc='', users=None, inherit_users=False, owner='')#

Replace the Quick Query with this new version. (*)

Only used for overwriting an existing query, not for writing a new one.

Parameters:
  • querypath (str) – Path to save the query to in the 1010data object tree.

  • title (str) – Title to give to the query (default "").

  • sdesc (str) – Short description (default "").

  • ldesc (str) – Long description (default "").

  • users (list(str)) – List of users to give access to. Default is None, which means make the query private unless inherit_users is True.

  • inherit_users (bool) – If users is None, then set the allowed users to inherit from the parent folder.

  • owner (str) – Set the owner of the folder to someone else. Default "", i.e., the current user.

Raises:

TentenException if the transaction fails.

See also

save()

rowasdict(self, n)#

Row at given index, presented as a dictionary. (*)

Same as {k : self[k][n] for k in self.coldict.keys()}

rowiterator(self, *args, **kwargs)#

Iterate over the rows of this query.

Arguments may be integers or strings, which are used to specify columns of the query (as index or colbyname). If none are specified, it’s taken to mean all the columns, in index order. The iterator returns a tuple of the values of all the specified columns, in the specified order.

This iterator is distinct and separate from ordinary access to the columns, and pulls down all the named columns together, instead of making a separate call for each column, so it will be more efficient if accessing a table row by row.

See also

rows

run(self)#

Execute query. Must be done before data can be accessed. (*)

Raises:

TentenException if the transaction fails.

save(self, querypath, title='', sdesc='', ldesc='', force=False, users=None, inherit_users=False, owner='')#

Save this query as a Quick Query. (*)

Parameters:
  • querypath (str) – Path to save the query to in the 1010data object tree.

  • title (str) – Title to give to the query (default "").

  • sdesc (str) – Short description (default "").

  • ldesc (str) – Long description (default "").

  • force (bool) – If there is already a query at the given querypath (i.e., saving as a new query failed), replace it with this one. Otherwise an exception is raised.

  • users (list(str)) – List of users to give access to. Default is None, which means make the query private unless inherit_users is True.

  • inherit_users (bool) – If users is None, then set the allowed users to inherit from the parent folder.

  • owner (str) – Set the owner of the folder to someone else. Default "", i.e., the current user.

Raises:

TentenException if the transaction fails.

saveFile(self, dest, targettype, names=False, headers=False, compression='zip')#

Save the results of a query to a file on FTP or cloud storage.

Parameters:
  • dest – A SourceInfo object with a single SourceFile in it, or a string, which is converted to one.

  • targettype (str) – A string (or bytes) specifying the format to save, currently one of “csv”, “xlsx”, “pdf”, “tde”, “parquet”.

  • names (bool) – Add a line at the top with the column names? Default False.

  • headers (bool) – Add a line at the top (below the names, if any) with column headers? Default False.

  • compression (str) – What kind of compression to use. Valid values are “zip” (default), “gzip”, and “none”.

Raises:

TentenException if the transaction fails.

saveTableMaterialize(self, *args, **kwargs)#

Call Session.saveTableMaterialize() on the Session object.

raise an Exception if this Query is not the ‘current’ one which would be saved.

saveToFTP(self, filename, sep='|', linesep='\n', namerow=False, headrow=False, compression='zip')#

Save this query to FTP. (*)

If namerow is True, first row will be taken to be column names. If headrow is True, first row (or second if namerow is also True) will be taken as column headers.

Parameters:
  • filename (str) – Name of FTP file to save.

  • sep (str) – Field separator. Default "|".

  • linesep (str) – Record separator. Default "\n".

  • namerow (bool) – Add a line at the top with the column names? Default False.

  • headrow (bool) – Add a line at the top (below the names, if any) with column headers? Default False.

  • compression (str) – What kind of compression to use. Valid values are “zip” (default), “gzip”, and “none”.

Raises:

TentenException if the transaction fails.

cursor#

Current index

ldesc#

Long description of the base table of this query. (*)

messagetext#

Session message text.

This is the session’s message text, which is filled in by CSDK layer, and serves as some documentation of the status of the last operation performed.

modelcols#

How to handle “model” columns.

If True, retrieve “model-type” columns as dictionaries. If False (default), model columns will all simply contain the string "MODEL".

nrows#

Total number of rows in the result

rawValues#

rawValues: object

rows#

Get a RowIterator over the rows of the query.

This is exactly the same as rowiterator(). Note: a new RowIterator is created for each access of this property.

Return type:

RowIterator

sdesc#

Short description of the base table of this query (*)

title#

Title of the base table of this query (*)

transfermode#

Get the transfer mode (compressed, binary, etc) for this query.

win_end#

The index of the end point of the currently loaded window (last fetch).

win_size#

The size of window to download (configurable)

Note: you can change win_size after reading in some values, so win_start+win_size does not necessarily equal win_end.

win_start#

The index of the starting point of the currently loaded window (last fetch)

class py1010.RowIterator(query, cols, start=0, stop=None, step=1, notuple=False, *)#

Indexable iterator over a query’s rows.

An object that represents a generator that yields rows, or multi-column tuples, of successive indexes through the query. It shares the columns with the query. Accessing the columns outside of the iterator at indices outside of the iterator’s current window may cause performance issues.

Construct RowIterator for query.

Creates a row iterator for the named query, starting at the given index (default zero); the rest of the arguments specify the columns, either by name or by index.

Parameters:
  • query (Query) – The Query object.

  • cols – A list of column numbers or names (can be mixed) specifying which columns are to be iterated over, and the order.

  • start (int) – Starting index. Default 0.

  • stop (int) – index. Default None, i.e. end of the query.

  • step (int) – Step size, default 1. Negative step values are not supported.

  • notuple (bool) – If True and there is only one column, yield the values “bare” and not wrapped in one-element tuples (as it normally would, since it’s considered a “row”).

convertindex(self, n)#

Convert an index based on the RowIterator’s slice info.

When an index of a slice is referenced, the slice’s information has to be factored in to determine the “base” index. That is, arr[10:50:5][2] is actually arr[20], and arr[10:50:5][15] is out of bounds, even if arr itself is long enough. A RowIterator is (or may be) a slice, so this translation has to happen.

Parameters:

n (int) – The index to convert.

Return type:

int

fetchrows(self, starting=None)#

Fetch a window of rows. Users should not need to call this. (*)

getrow(self, n)#

Get a tuple of data at specified index. (*)

Returns a tuple of the elements in this iterator’s columns at a given index. If the index is not available from the relevant rows, it fetches the window of rows. Users should not be calling this, but should be using next().

Parameters:

n (int) – Index to get.

Returns:

Tuple of values of columns at the index.

Raises:

IndexError if n is greater than the number of rows in the query.

Raises:

TentenException if the transaction fails.

has(self, n)#

Is the given index currently loaded?

Checks the windows of all the columns in the iterator to make sure the index is available for all of them. Columns may have different windows or may be accessed outside the iterator, but has() only returns True if the given index is in the windows of all the columns.

class py1010.Session(url, username, password, logintype, group=None, retry=1073741824, wait=10, logfile=None, mode='w', oktadesc=None, authentication=None, *, authscript=None, **kwargs)#

Object representing a 1010 session.

Create with 1010 URL, username (or group ownername), password, and type of login (py1010.KILL, py1010.NOKILL, or py1010.POSSESS). If this is a SAM pool (group) login, add another parameter for the groupID. py1010 automatically acquires a UID and logs in with it.

Attempting to use a closed session, either directly or indirectly by using a Query that was created for it, raises a TentenClosedSessionException.

Properties: (default value in parens. (w) means the value is writable, and (w-) means the value is writable but not readable)):

messagetext

Last message text from 1010 API

rc

Return code (int) of last operation performed

timeout

Timeout parameter tuple: (connectTimeout, timeout) (w)

reuseConnection

Reuse the same connection (True) (w-)

lenient

Treat non-conforming columns leniently (False) (w-)

ignoreSSLErrors

No exception on SSL errors (False) (w-)

APIversion

API version of this session (w)

systemVersion

System version this session is logged into

lastResponse

Last response from the session

transactions

Raw bitmask of available transactions

dbmdo

Used for calling the dbm() method with a slightly different syntax. session.dbmdo.endpoint(key1=val1, ...) is equivalent session.dbm("endpoint", key1=val1, ...), with one or two special cases (see the docstrings on DBMDo).

Create or connect to a 1010 session (*)

Create or connect to a 1010 session with the specified URL, username (or owner name for SAM Pool login), password, login type (py1010.KILL, py1010.NOKILL, or py1010.POSSESS), and group name (for SAM pool login). For SAM pool logins, a maximum number of retries and the number of seconds to wait between retries can also be specified (silently increased if below the minimum of 10s.)

If the password supplied is None, py1010 will prompt the user for it.

You can also create a Session object without logging in if you have the SessionID (SID) and the Encrypted Password (EPW) of an existing 1010data session. Pass in the url and username, and pass in the EPW as the password. Use py1010.POSSESS as the logintype, and add the keyword-only parameter “sid=SID” to pass in the session ID.

For SSO login or other external authentication, use the keyword-only parameter authentication=TAG, with TAG being a string specifying the style of authentication. See your administrator for your organization’s custom tag, if one is being used.

Parameters:
  • url (str) – The 1010data gateway URL.

  • username (str) – Username on 1010data or SSO, or SAM-pool owner ID.

  • password (Optional[str]) – Password for 1010data or SSO.

  • logintypepy1010.POSSESS to use an existing session or start a new one; py1010.KILL to end an existing session (if any) before starting this one; py1010.NOKILL to error if a session already exists.

  • group (str) – SAM-pool group name, if applicable.

  • retry (int) – Number of times to retry if no SAM-pool user-id is available (default no limit).

  • wait (int) – Seconds to wait between retries for a user-id.

  • logfile – Filename (or python filehandle) for writing XML log.

  • mode (str) – Mode for writing logfile (“a” for appending, “w” (default) for truncating.)

  • oktadesc (OktaLogin) – OktaLogin object for Okta logins.

  • authentication (str) – (keyword-only) Authentication tag for SSO logins.

  • authscript (str) – (keyword-only) External authentication script for SSO logins, if not the standard one.

Returns:

Session object.

addKey(self, keyobj, key='', id_='', region='')#

Add a key to the server-side keystore.

Add a new AWS key to the keystore under a user-chosen name.

Parameters:

keyobj (AWSKey) – The AWSKey object to add

OR you can run with just the information for creating the AWSKey object and the function will construct it for you

Parameters:
  • keyobj (str) – The user-defined name (a string) for the key.

  • key (str) – The AWS secret key.

  • id_ (str) – The AWS id.

  • region (str) – The AWS region.

Raises:

TentenTransactionException if a key with this name already exists in the keystore (among other reasons.)

addTable(self, spec, sync=False)#

Add a table from data already uploaded to FTP account.

Parameters:
  • spec (str) – XML string containing the specifications of the data.

  • sync (bool) – Should the upload be synchronous (True) or asynchronous (False, default)?

Returns:

The name of the table created.

Raises:

TentenException if the transaction fails.

addTableEnd(self, table)#

Close and commit a transaction addtab.

Run this method at the end of a “transactional” addtab process to indicate that you are finished uploading chunks of data and to close and save the table. Equivalent to sending a zero-byte chunk, i.e. self.addTableFeed(table, b'', False)

Parameters:

table (str) – Name of the table to finish.

Raises:

TentenException if the transaction fails.

addTableFeed(self, table, data, compress=True)#

Send a chunk of data in a transactional addtab process.

In a “transactional” addtab process, you run addTableSpecs() with a source spec which includes no SourceFiles. Then you use this method to send chunks of data to be added to the table, and finally call addTableEnd() to close and commit the table.

Parameters:
  • table (str) – Name of the table you are uploading to.

  • data (bytes) – The chunk of data to send, formatted as described in the SourceSpec.

  • compress (bool) – Whether or not to compress the data being sent (default True)

Raises:

TentenException if the transaction fails.

addTableSpecs(self, sourcespec, tablespec, sync=False)#

Add a table using given specifications.

Runs the API’s addtable transaction. To initiate a “transactional” addtable, in which you upload the data in chunks, use a SourceSpec that lists no files.

Parameters:
  • sourcespec (SourceSpec) – A SourceSpec object giving the location and structure of the source file(s) and the columns therein.

  • tablespec (TableSpec) – A TableSpec object giving the name and other metadata for the target table, as well as the columns there. If no columns are supplied here, the table column information will be copied from the source column information.

Raises:

TentenException if the transaction fails.

addTableStatus(self)#

Check the status of an asynchronous upload with addTable()

Returns a tuple (status, numrecs). The status is an integer with the following meanings:

0:
  • TENTEN_ADDTAB_IDLE

1:
  • TENTEN_ADDTAB_FAILED

2:
  • TENTEN_ADDTAB_INITIALIZING

3:
  • TENTEN_ADDTAB_LOADING

4:
  • TENTEN_ADDTAB_COMPLETED

5:
  • TENTEN_ADDTAB_DIAGNOSED

The numrecs is the number of records (rows) uploaded so far (or in total if the upload is complete).

Raises:

TentenException if the transaction fails.

addTableStatusList(self)#

Get a list of the status of addtable transactions.

Returns:

A list of AddtableStatus objects for all of the addtable transactions that have been started so far in this session.

Raises:

TentenException if the transaction fails.

api2(self, method, data=None, *, form='json', ignoreJSONErrors=False)#

Call the QuickApp API.

Parameters:
  • method (str) – The API (QuickApp) to call.

  • data – If given, call with be POST with this data (json-encoded)

  • form (str) – (keyword-only) Request the data in a given format. Default is “json”; other options may include “html” or “text”; the actual possibilities are determined by the server-side code.

  • ignoreJSONErrors (bool) – (keyword-only) If json.loads (or decode('utf-8')) fails to decode the response, if this parameter is True, just return the response data instead of raising an exception.

Returns:

Decoded json, or plain bytes from server if form is not “json”. If form is “json” and ignoreJSONErrors is True, returns the decoded json unless there is an error either in decoding the bytes to a string or in decoding the json, in which case it returns the plain bytes.

authenticateCredentials(type cls, url, user, password)#

Check that the given credentials are valid. (*)

This does not require an open session (class method) and does not create one.

Returns:

True if the credentials are valid.

autoSpec(self, data)#

Run the API’s autospecfromdata transaction on the given data.

Returns a dictionary of sourcefile information, with the source columns as dictionaries in a list under ‘columns’.

The autoSpecFile() method is probably superior in most situations.

Parameters:

data (str) – Sample data (string) to be specced.

Returns:

A dictionary of inferred specs.

Raises:

TentenException if the transaction fails.

autoSpecFile(self, spec=None)#

Run the API’s autospec transaction on the source information provided.

Parameters:

spec – A SourceSpec object, or something that can be converted into one (a string or list of strings, a SourceFile object or list of SourceFiles, or a SourceInfo). It may also contain source columns and other information to be used as “hints” by the auto-spec. Pass None to start from scratch.

Returns:

Inferred SourceSpec.

Return type:

SourceSpec

Raises:

TentenException if the transaction fails.

clearCache(self)#

Clears the 1010data cache. (*)

close(self)#

Close the session and release userID if necessary.

Closed sessions can no longer be accessed for most purposes; attempting to do so raises a TentenClosedSessionException.

createDirectory(self, pathname, title=None, users=None, uploaders=None, inherit_users=False, inherit_uploaders=False)#

Run Directory.createDirectory() (q.v.)

dbm(self, trans, **kw)#

Call the ‘dbm’ API2 endpoint.

Nearly equivalent to self.api2('dbm', data={'cmd': [trans, kw]}).

Parameters:
  • trans (str) – The dbm transaction to invoke

  • **kw – Key-value arguments for the given transaction

Returns:

The transaction’s response, decoded from json

Raises:

DBMError if the transaction fails.

debugAccum(self, on=True)#

Switch ACCUM (server) into (or out of) Debug mode.

delKey(self, keyobj, key='', id_='', region='')#

Remove a key from the server-side keystore.

Remove an AWS key in the keystore, specified by name.

Parameters:

keyobj (AWSKey) – The AWSKey object containing the name of the key you want to remove. The other information in the object is ignored.

OR you can run with just the information for creating the AWSKey object and the function will construct it for you

Parameters:
  • keyobj (str) – The user-defined name (a string) for the key.

  • key – (optional, ignored)

  • id_ – (optional, ignored)

  • region – (optional, ignored)

Raises:

TentenTransactionException if a key with this name does not already exist in the keystore (among other reasons.)

directory(self, dirname)#

Returns a Directory object describing the directory named (*)

Parameters:

dirname (str) – Pathname of directory

Return type:

Directory

disableLog(self)#

Disable XML logging

dropTable(self, table)#

Delete a table (*)

Parameters:

table (str) – Pathname of table to drop.

Raises:

TentenException if the transaction fails.

enableLog(self, filename='log.log', mode='a')#

Enable XML logging.

Parameters:
  • filename (str) – Filename of log.

  • mode (str) – Mode for writing ("a" or "w". Default "a")

getQueryOps(self, pathname)#

Get the ops of a QuickQuery.

Returns the ops (the XML macro-code) of a saved QuickQuery in the 1010data object tree.

Parameters:

pathname – The path in the object tree to the saved QuickQuery.

Returns:

A dictionary {'base': basename, 'ops': optext}, containing the base table of the QuickQuery (if any; None if there is none) and the text of the ops saved in the query.

Raises:

TentenException if the transaction fails.

getRetry(self)#

Get retry parameters.

Returns:

A tuple: (retryMax, retryTimeMin, retryTimeRange, retryBase)

getUID(type cls, url, owner, password, group)#

DEBUG ONLY: acquire a UID from a group. Leaves UID unused, claimed, and unreleasable!

logRawXML(self)#

Enable raw XML logging

memUsage(self)#

Returns the memory usage of the session, in bytes (*)

Return type:

int

mergeTablesMaterialize(self, tablelist, destpath, title=u'', sdesc=u'', ldesc=u'', link=u'', int maxdown=-1, int replace=False, users=None, char *segmentation=NULL, int sortseg=False, segbyAdvise=None, sortsegAdvise=None, int timeSeries=False, links=None, char *sort=NULL, *, inherit_users=False)#

Merge a list of tables. (*)

Runs the CSDK tenten_MergeTablesMaterialize function. This saves the most recently-run Query.

Parameters:
  • tablelist (list(str)) – List of table-names to be merged

  • destpath (str) – Path in the 1010 object tree to save this table.

  • title (str) – Title of the table (default "")

  • sdesc (str) – Short description (default "")

  • ldesc (str) – Long description (default "")

  • link (str) – String to be prepended to titles of linked columns (default "")

  • maxdown (int) – Maximum number of cells that can be downloaded at once. Default -1, for no limit

  • replace (bool) – Replace the table if it already exists (default False)

  • users (list(str)) – List of usernames with access (default None)

  • segmentation (str) – Comma-separated string, a list of columns to segment on (Default None)

  • sortseg (bool) – Use SortSeg (True) or SegBy (False, default)

  • segbyAdvise (list(str)) – List of comma-separated column-name sequences which can be assumed to be segmented with SegBy segmentation if segmentation is performed (Default None)

  • sortsegAdvise (list(str)) – List of comma-separated column-name sequences which can be assumed to be segmented with SortSeg segmentation if segmentation is performed (Default None)

  • timeSeries (bool) – Generate time-series metadata (default False)

  • links (list(str)) – A list of 1010 link ops (default None)

  • sort (str) – Comma-separated string, a list of columns by which to sort the contents of each segment (default None)

  • inherit_users (bool) – (keyword-only): If True, ignore the list of users given (if any) and set the user permissions to ‘inherit’.

Raises:

TentenException if the transaction fails.

moveTable(self, oldpath, newpath)#

Move a table from one pathname to another (*)

Parameters:
  • oldpath (str) – Old pathname.

  • newpath (str) – New pathname.

Raises:

TentenException if the transaction fails.

peekOnce(self, key, mode)#

Get the value of a server-side variable

Parameters:
  • key – Content of <peek> tag to send to server

  • mode – Value of attribute <peek mode> to send to server

Returns:

Value of the variable.

Return type:

bytes

ptr(self, char *name=NULL)#

Encapsulated C pointer. Internal use only.

putKey(self, keyobj, key='', id_='', region='')#

Change a key on the server-side keystore.

Alter an AWS key in the keystore, specified by name, to contain new information.

Parameters:

keyobj (AWSKey) – The AWSKey object with the name and updated info

OR you can run with just the information for creating the AWSKey object and the function will construct it for you

Parameters:
  • keyobj (str) – The user-defined name (a string) for the key.

  • key (str) – The new AWS secret key.

  • id_ (str) – The new AWS id.

  • region (str) – The new AWS region.

Raises:

TentenTransactionException if a key with this name does not already exist in the keystore (among other reasons.)

query(self, table, xml='')#

Creates a query object for this session, for a given table and XML.

You can also pass an existing query object (from this or another session) to clone it in this session. The query is not run.

Parameters:
  • table (str) – Name of the table for this query.

  • xml (str) – XML code of the query. Default ''.

Return type:

Query

readKeys(self)#

Get a list of keys in the server-side keystore

Returns:

a list of AWSKey objects

releaseUID(self)#

DEBUG ONLY: release a claimed UID

relog(self, password, logintype=-1)#

Get a new Session object based on this one. (*)

Returns a NEW session object, logging in again with the same URL and username, and the given password and logintype (default POSSESS). Other configurable properties are not carried over into the new session. May be called on a closed session.

Parameters:
  • password (Optional[str]) – Password for account; prompt for password if None.

  • logintype – py1010.POSSESS (default), py1010.KILL, or py1010.NOKILL.

Return type:

Session

saveTableMaterialize(self, tablepath, title=u'', sdesc=u'', ldesc=u'', link=u'', int maxdown=-1, int replace=False, int append=False, int appendable=True, int temporary=False, users=None, char *segmentation=NULL, int sortseg=False, segbyAdvise=None, sortsegAdvise=None, int timeSeries=False, links=None, char *sort=NULL, *, inherit_users=False)#

Create, replace, or append to a table. (*)

Runs the CSDK tenten_SaveTableMaterialize function. This saves the most recently-run Query.

Parameters:
  • tablepath (str) – Path in the 1010 object tree to save this table.

  • title (str) – Title of the table (default "")

  • sdesc (str) – Short description (default "")

  • ldesc (str) – Long description (default "")

  • link (str) – String to be prepended to titles of linked columns (default "")

  • maxdown (int) – Maximum number of cells that can be downloaded at once.Default -1, for no limit

  • replace (bool) – Replace the table if it already exists (default False)

  • append (bool) – Append to the end of an existing table (default False)

  • appendable – Allow this table to be merged with other tables (default False)

  • temporary (bool) – Make this table temporary (default False)

  • users (list(str)) – List of usernames with access (default None)

  • segmentation (str) – Comma-separated string, a list of columns to segment on (Default None)

  • sortseg (bool) – Use SortSeg (True) or SegBy (False, default)

  • segbyAdvise (list(str)) – List of comma-separated column-name sequenceswhich can be assumed to be segmented with SegBy segmentation if segmentation is performed (Default None)

  • sortsegAdvise (list(str)) – List of comma-separated column-name sequenceswhich can be assumed to be segmented with SortSeg segmentation if segmentation is performed (Default None)

  • timeSeries (bool) – Generate time-series metadata (default False)

  • links (list(str)) – A list of 1010 link ops (default None)

  • sort (str) – Comma-separated string, a list of columns by which to sort the contents of each segment (default None)

  • inherit_users (bool) – (keyword-only): If True, ignore the list of users given (if any) and set the user permissions to ‘inherit’.

Raises:

TentenException if the transaction fails.

searchTables(self, filter=None, maxresults=0)#

Search for tables with matching names. (*)

Parameters:
  • filter (Optional[str]) – String to search for. Default None.

  • maxresults (int) – Maximum number of results to return. Default 0, i.e., no maximum.

Returns:

A list of DirEntry objects.

Return type:

list(DirEntry)

Raises:

TentenException if the transaction fails.

setRetry(self, long retryMax, long retryTimeMin, long retryTimeRange, double retryBase)#

Set retry parameters.

Parameters:
  • retryMax (int) –

  • retryMin (int) –

  • retryTimeRange (int) –

  • retryBase (float) –

Raises:

TentenException if the transaction fails.

setUserAgent(self, agent, password)#

Set the UserAgent string. For internal use only.

setXFF(self, xff)#

Set X-Forwarded-For string

Parameters:

xff (str) – String to set X-Forwarded-For to.

stop(self)#

Send a stop transaction

updateObject(self, dirname)#

Refreshes this session’s view of a file or a directory (*)

Recursively refreshes subdirectories.

Parameters:

dirname (str) – Pathname of directory.

Raises:

TentenException if the transaction fails.

uploadXMLTable(self, char *xmlbuf)#

Upload raw XML as a table (*)

Parameters:

xmlbuf (str) – String containing the XML to be uploaded.

Raises:

TentenException if the transaction fails.

warmPool(type cls, url, owner, password, group, queries=None, logfile=None)#

Calls warmpool()

Class method.

APIversion#

The API version of this session (Configurable)

epw#

The encrypted password of this session.

ignoreSSLErrors#

Determines if you get an exception on SSL errors or ignore them

Default: False

Type:

bool

lastResponse#

The last response from the session

messagetext#

Message text of the session.

pswd#

PSWD of a session

rc#

The return-code (integer) of the last transaction performed

realurl#

realurl: object

reuseConnection#

Determines if the same connection is reused between transactions Default: True

sessionid#

SID of a session

sid#

The session ID of this session.

systemVersion#

The system version this session is logged into

timeout#

Connection timeout parameters.

Settable, using the same tuple form.

Returns:

A tuple: (connectTimeout, timeout)

transactions#

The raw bitmask of available transactions

url#

url: object

userAgent#

The user agent for this session

username#

username: object

class py1010.SourceColumnInfo(name=None, title=None, type_=None, format=None, int width=0, exp=None, double scale=0.0, int alpha=Alpha.SKIP, int order=0, int skip=Skip.NOSKIP, int nowrite=Write.WRITE, ptr=None, *)#

Class for holding metadata about a column in a source to be uploaded.

Several of the fields are meant to hold values from special enumeration classes, which are internal classes of SourceColumnInfo. So the nowrite parameter can be SourceColumnInfo.Write.WRITE or SourceColumnInfo.Write.NOWRITE. See below, and individual docstrings.

Properties:

name

Column name

title

Column title

type_

Type of column: a string (or bytes): “text”, “int”, “float”, or “bigint”

format

Column format descriptor (string)

width

Width of column; 0 (default) for no input width.

exp

Expression to be applied to this column before upload

scale

Decimal value by which to divide this column before upload. 0.0 (default) for none

alpha

Alphabetic case into which to force this column before upload. One of SourceColumnInfo.Alpha.UPPER, SourceColumnInfo.Alpha.LOWER, or SourceColumnInfo.Alpha.SKIP (default)

order

Positive integer for the position of this column in a reordering; 0 (default) for no reordering.

skip

Skip this column or not? One of SourceColumnInfo.Skip.SKIP or SourceColumnInfo.Skip.NOSKIP (default)

nowrite

Write this column? One of SourceColumnInfo.Write.WRITE (default) or SourceColumnInfo.Write.NOWRITE.

class Alpha(value)#

Enumeration for alpha conversion:

  • Alpha.UPPER

  • Alpha.LOWER

  • Alpha.SKIP

class Skip(value)#

Enumeration for skipping columns:

  • Skip.SKIP

  • Skip.NOSKIP

class Write(value)#

Enumeration for writing columns:

  • Write.WRITE

  • Write.NOWRITE

alpha#

Case transformation.

Alphabetic case into which to force this column prior to upload: SourceColumnInfo.Alpha.UPPER, SourceColumnInfo.Alpha.LOWER, or SourceColumnInfo.Alpha.SKIP.

exp#

Column expression.

format#

Column format.

name#

Name of the column.

nowrite#

Whether or not to write this column.

Set to SourceColumnInfo.Write.WRITE or SourceColumnInfo.Write.NOWRITE

order#

A positive integer indicating this column’s position in a revised column order, or 0 for no reordering.

scale#

Column scale.

Decimal value by which to divide the values in this column prior to upload, or 0 for none.

skip#

Whether or not to skip this column on loading.

Set to SourceColumnInfo.Skip.SKIP or SourceColumnInfo.Skip.NOSKIP

title#

Column title.

type#

Type of column.

A string (or bytes), one of: “text”, “int”, “float”, “bigint”

width#

Column width.

class py1010.SourceFile(path, bucket=None, keyname=None, sheetID=None, range=None, account=None, container=None, ptr=None, *)#

Class describing an individual file as a data source.

A “source” for 1010data is a description of some file outside of the 1010data object tree. A source is described by a SourceInfo object, which specifies features like format and column-separators, etc, and a SourceInfo contains one or more SourceFile objects, which specify locations of actual files (FTP upload directories or cloud storage services.)

Since a “source” describes an external resource, the same objects are used as “destinations” to describe files and formats for writing output.

Construct a SourceFile object.

The SourceFile contains location information for a file outside of 1010 (in an FTP upload directory, an S3 bucket, or in ABS).

Parameters:
  • path (str) – The filename of the file.

  • bucket (str) – The S3 bucket, for files in S3. Leave as None for files in FTP or ABS.

  • keyname (str) – The name assigned to the AWS key to use to access the file. See the Session.addKey() method of Session objects. Leave as None for files in FTP.

  • sheetID (str) – To specify a worksheet in an XLSX workbook, pass the sheet’s ID here (as returned by the getworksheets transaction).

  • range (str) – For specifying a cell-range in an XSLX worksheet.

  • account (str) – The ABS account to be used. Leave as None for files in FTP or S3 storage.

  • container (str) – The ABS container to be used. Leave as None for files in FTP or S3 storage.

init(self, path, bucket, keyname, sheetID, range, account, container)#

Set object attributes on construction.

account#

The account to use on ABS to access the file, or None.

bucket#

The S3 bucket containing the file (or None).

container#

The container to use on ABS to access the file, or None.

keyname#

The user-assigned name of the AWS key to use to access the file, or None

path#

The filename of the file.

range#

A range of cells in an XLSX worksheet which this object refers to, if relevant.

(Implementation note: this property will not contain the value None. If you set it to None, that really sets it to b'')

sheetID#

The sheetID of the worksheet this object refers to, within an XLSX workbook, if relevant.

(Implementation note: this property will not contain the value None. If you set it to None, that really sets it to b'')

class py1010.SourceInfo(files=None, rectype=None, sep=None, eor=None, maskw=None, mchr=None, arch=None, format=None, long begbytes=0, long begrecs=0, long numrecs=0, int autoCorrect=0, int numCols=0, ptr=None, *)#

Class describing an individual file as a data source.

A “source” for 1010data is a description of some file outside of the 1010data object tree. A source is described by a SourceInfo object, which specifies features like format and column-separators, etc, and a SourceInfo contains one or more SourceFile objects, which specify locations of actual files (FTP upload directories or cloud storage services.)

Since a “source” describes an external resource, the same objects are used as “destinations” to describe files and formats for writing output.

SourceInfo objects contain metadata about the format of an external file. Several of the fields are meant to hold values from special enumeration classes, which are internal classes of SourceInfo. So the rectype can be SourceInfo.RecType.SEPARATED or SourceInfo.RecType.FIXED. See below, and individual docstrings.

Properties:
sourceType

Type of source (FTP, S3, etc.)

sep

Column separator

eor

Record separator

maskw

Max length of variable-width columns

mchr

“Masking” character

arch

Architecture: little-endian or big-endian

format

Type of file (“xlsx” or empty)

begbytes

Number of bytes to skip at the start

begrecs

Number of records to skip at the start

numrecs

Number of records to upload (0 for all)

autoCorrect

Enable simple autocorrection feature?

truncate

Autocorrect truncate control

pad

Autocorrect pad control

fix_mask

Autocorrect fix-mask control

numCols

Number of columns

ignoreNull

Replace '\\0' with ‘ ‘?

Constructor for SourceInfo objects.

Parameters:
  • files – A list of SourceFile objects (or strings, which are taken to be filenames in an FTP directory)

  • rectype – Record type: RecType.SEPARATED or RecType.FIXED or None

  • sep – Column separator

  • eor – Record separator

  • maskw – Max width of variable columns

  • mchr – “Masking” character

  • arch – Architecture: Arch.BENDIAN or Arch.LENDIAN or None

  • format – Either “xlsx” or None (default) for text files

  • begbytes – Bytes to skip at the beginning (default 0)

  • begrecs – Records to skip at the beginning (default 0)

  • numrecs – Number of records to load (default 0, for “all”)

  • autoCorrect – Enable autoCorrect? (default 0 (False))

  • numCols – Number of columns

class Arch(value)#

Enumeration for architecture (endianness):

  • Arch.LENDIAN (Little-endian)

  • Arch.BENDIAN (Big-endian)

class AutoCorrectType(value)#

Enumeration for auto-correction type:

  • AutoCorrectType.NONE

  • AutoCorrectType.RIGHT

  • AutoCorrectType.LEFT

  • AutoCorrectType.LONG

  • AutoCorrectType.SHORT

class RecType(value)#

Enumeration for record type:

  • RecType.SEPARATED

  • RecType.FIXED

class SrcType(value)#

Enumeration for source type:

  • SrcType.FTP

  • SrcType.S3

  • SrcType.CHUNK

  • SrcType.ABS

getWorksheets(self, Session s)#

Run the getworksheets transaction on this SourceInfo object

This source object should describe a .xlsx source.

Parameters:

session (Session) – Session object to use to run getworksheets.

Returns:

A list of dictionaries of the form {"name": NAME, "id": ID} for all the worksheets.

Return type:

list(dict)

Raises:

TentenException if the transaction fails.

init(self, files=None, rectype=None, sep=None, eor=None, maskw=None, mchr=None, arch=None, format=None, long begbytes=0, long begrecs=0, long numrecs=0, int autoCorrect=0, int numCols=0)#

Initialize fields on construction.

arch#

Architecture or “endianness” of this source. May be Arch.BENDIAN (big-endian) or Arch.LENDIAN (little-endian) or None (unspecified).

autoCorrect#

Specify “simple” autocorrection: True or False.

begbytes#

Bytes to skip at the beginning.

begrecs#

Records to skip at the beginning.

eor#

Row separator for this Source.

files#

files: object

filter_target#

filter_target: object

fix_mask#

Autocorrect fix-mask control, for delimited columns only.

Set to AutoCorrectType.NONE, AutoCorrectType.LEFT, AutoCorrectType.RIGHT, AutoCorrectType.LONG, or AutoCorrectType.SHORT.

format#

File-format of this Source. May be None or “” (for text files) or “xslx”.

ignoreNull#

Replace NUL (’\0’) characters with spaces? Set to True, False, or None (unspecified, default).

maskw#

The “masking width,” or the maximum width of variable-length columns in this source. Default 10000.

mchr#

Masking character for this Source.

numCols#

Number of columns in input data.

numFiles#

Number of SourceFiles in this Source. This property may not be set directly.

numrecs#

Number of records to read.

pad#

Autocorrect pad control.

Set to AutoCorrectType.NONE, AutoCorrectType.RIGHT, or AutoCorrectType.LEFT.

rectype#

Record type for this Source.

May be RecType.SEPARATED or RecType.FIXED or None (unspecified).

sep#

Column separator for this Source.

sourceType#

Type of this source.

May be SrcType.S3, SrcType.ABS, or SrcType.FTP. This property is not set directly, but is determined by whether or not the first SourceFile has a bucket property or container property set. (SourceFiles of different types may not be combined in the same SourceInfo.)

truncate#

Autocorrect truncate control.

Set to AutoCorrectType.NONE, AutoCorrectType.RIGHT, or AutoCorrectType.LEFT.

class py1010.SourceSpec(self, source=None, sourceCols=None)#

A class to hold both SourceInfo and a list of SourceColumnInfo.

toXML(self, session)#

Convert to XML representation.

class py1010.StructWrapper(magic, clazz, ptr=None, *)#

INTERNAL USE ONLY!

Wrapper for C-level structures. Works similarly to the Java2 SDK; see the comments there for description.

NEVER USE THIS! Create objects with the _New class method.

wrapPtr(type cls, ptr, clazz)#

Internal use only!

Never use this. Only use the _New class method.

PTR#

NOT for use by users!

class py1010.SyncInfo(*args, **kwargs)#
class py1010.TableInfo(name, int ID=0, title=None, sdesc=None, ldesc=None, type_=u'', int secure=0, int own=0, owner=None, update=None, int favorite=0, users=None, display=None, int report=0, int chart=0, link=u'', long numRows=0, long numBytes=0, int segs=0, int access=0, long maxdown=0, mode=Mode.REPLACE, stripe=None, stripe_factor=None, ptr=None, *)#

Class for holding metadata about a table as an upload target.

Holds data about a table for uploading (with Session.addTableSpecs()).

Several of the fields are meant to hold values from special enumeration classes, which are internal classes of TableInfo. So the mode can be TableInfo.Mode.APPEND or TableInfo.Mode.REPLACE or TableInfo.Mode.NOREPLACE. See below, and individual docstrings.

class Mode(value)#

Enumeration for mode:

  • Mode.APPEND

  • Mode.REPLACE

  • Mode.NOREPLACE

class Perm(value)#

Enumeration for permissions:

  • Perm.PRIVATE

  • Perm.INHERIT

class SegType(value)#

Enumeration for segmentation type:

  • SegType.SEGBY

  • SegType.SORTSEG

class TimeSeries(value)#

Enumeration for time-series:

  • TimeSeries.TS

  • TimeSeries.NOTS

init(self, name, int ID=0, title=None, sdesc=None, ldesc=None, type_=u'', int secure=0, int own=0, owner=None, update=None, int favorite=0, users=None, display=None, int report=0, int chart=0, link=u'', long numRows=0, long numBytes=0, int segs=0, int access=0, long maxdown=0, mode=Mode.REPLACE, stripe=None, stripe_factor=None)#
access#

Boolean 1 or 0 indicating whether or not this table is accessible.

chart#

Boolean 1 or 0 indicating whether or not chart specifications are saved for this table.

favorite#

Boolean 1 or 0 indicating whether or not the transaction UID has favorited this table.

id#

Unique identifier for this table.

ldesc#

Long description of the table, if any.

Link header of table, or NULL for no link header.

materialize#

Boolean 1 or 0 indicating whether or not this table is materialized.

maxdown#

Maximum download limit of table, or a non-positive integer for the default maxdown.

merge#

Boolean 1 or 0 indicating whether or not this table is appendable.

method#

Materialize method, or None for the default method.

mode#

Append or replace?

name#

Full path to the table.

numBytes#

Number of bytes in the table.

numCols#

Number of columns in this table.

numRows#

Number of rows in the table.

own#

Boolean 1 or 0 indicating whether or not the transaction UID is the owner of this table.

owner#

UID or groupname of the owner of this table, or None for the default owner.

report#

Boolean 1 or 0 indicating whether or not report specifications are saved for this table.

responsible#

Boolean 1 or 0 indicating whether or not the user is responsible for replication of data.

sdesc#

Short description of the table, if any.

secure#

Boolean 1 or 0 indicating whether or not this table is secure. Deprecated in API.

segmentation#

Comma-separated list of the names of segmentation columns.

segs#

Number of segments spanned by this table.

segsize#

Size of the segments of this table.

segtype#

Integer representing segmentation type of this table. Either TableInfo.SegType.SEGBY or TableInfo.SegType.SORTSEG. 0 if “segmentation” is None.

sort#

Comma-separated list of the names of sort columns.

stripe#

How many machines to stripe the data across.

stripe_factor#

Fraction of machines to stripe data across.

timeSeries#

Integer representing whether or not time-series segmentation is used for this table. Either TENTEN_TS or TENTEN_NOTS. 0 if “segmentation” is None.

title#

Title of the table, if any.

type#

Type of table. Currently, can be “REAL”, “VIEW”, “PARAM”, “MERGED”, “UQ”, or “TOLERANT”.

update#

Datetime of last modification to this table.

users#

users: object

class py1010.TableSpec(self, table=None, cols=None)#

Class to hold a TableInfo and a list of ColumnInfo.

toXML(self, Session session)#

Convert to XML notation.

class py1010.TentenObject#

The wrapper object type for 1010data Insights Platform objects

tentenentering(self, fun, *args, **kwargs)#

(log entering)

session#

session: py1010.Session

py1010.by(s)#

(internal function: string2bytes for python3)

py1010.bytes2string(dat, *args)#
py1010.entering(hsh, cls, fun, *args, **kwargs)#

(internal function: log entering function)

py1010.except_msg(msg)#

(decorator for exceptions)

py1010.except_zero(func)#

(decorator to trap failure (zero) and raise exception)

py1010.except_zero2(func)#

(decorator to trap failure (zero) and raise exception)

py1010.exiting(hsh, cls, fun)#

(internal function: log exiting function)

py1010.flushpool(url, owner, password, group, logfile=None, mode='w')#

Flush a SAM Pool. (*)

Invokes the “markgid” or “FlushPool” transaction on the group. This guarantees that any subsequent UIDs returned for the group haven’t been logged in before on a session and are fresh.

Parameters:
  • url (str) – the URL of the 1010 connection

  • owner (str) – username of the group owner

  • password (str) – the password for the group

  • group (str) – the name of the group (pool) to be flushed

  • file (log) – Optional name of file used for logging this transaction

  • mode (str) – fopen-style mode for opening logfile, defaults to “w”

Raises:

TentenException if the transaction fails.

py1010.initialize(force=False)#

Initialize 1010 API. You should not need to call this; creating a Session does this for you.

py1010.isfile(f)#
py1010.logit(func)#

(decorator for logging)

py1010.resetpool(url, owner, password, group, logfile=None)#

Release a SAM Pool’s IDs. (*)

Invokes the “resetpool” transaction on the group. This releases all the group’s UIDs.

Parameters:
  • url (str) – the URL of the 1010 connection

  • owner (str) – username of the group owner

  • password (str) – the password for the group

  • group (str) – the name of the group (pool) to be reset

  • logfile (Optional[str]) – filename for logging, if any.

Raises:

TentenException if the transaction fails.

py1010.sampoolstatus(url, owner, password, group, logfile=None)#

Check the status of a SAM pool.

Run the “samstatus” transaction on the given SAM pool and return the results.

Parameters:
  • url (str) – The url of the 1010data gateway.

  • owner (str) – The userid of the owner of the SAM pool.

  • password (str) – The owner’s password.

  • group (str) – The name of the group.

  • logfile (Optional[str]) – Filename to log the transaction (optional).

Returns:

a list of tuples of the form: [("groupname", "username", free, loggedin, marked)...] The groupname is the same for all the users. The “free”, “loggedin”, and “marked” values are boolean values for whether that userID is free, logged in, and/or marked.

py1010.st(s)#

(internal function: bytes2string for python3)

py1010.startLogging(filename='Py1010.log')#

Starts API-level logging.

py1010.stopLogging()#

Close logging.

py1010.teardown(force=False)#

End and clean up 1010 API.

py1010.twoparams(fun)#
py1010.warmpool(url, owner, password, group, queries=None, logfile=None)#

Warm a SAM Pool.

Parameters:
  • url (str) – The URL of the 1010 connection

  • owner (str) – Username of the group owner

  • password (str) – Password for the group

  • group (str) – Name of the group (SAM pool)

  • queries (str) – List of BaseQuery objects

  • logfile (Optional[str]) – File to log transactions to (default None)

Returns:

the number of IDs warmed.

Return type:

int

Raises:

TentenException if the transaction fails.