Shared Access Management pools
=================================
Shared Access Management (SAM) pools enable a single set of credentials to be shared between client side threads to leverage multiple threads of parallelism on the 1010data Insights Platform.
Creating a new :class:`py1010.Session` object for a SAM pool user is similar to creating a single-user ``Session`` object.
.. code-block::
testSession=py1010.Session(gateway, owner, pwd, py1010.POSSESS, group)
The SAM pool user session takes two different parameters. Instead of the individual user name, this ``Session`` object takes the ``owner`` parameter. The ``owner`` parameter is the group owner's user name. The ``group`` parameter is the group ID. The SAM pool ``Session`` constructor takes the following parameter variables:
* ``gateway`` - a 1010data Insights Platform URI
* ``group`` - a group ID
* ``owner`` - the group owner ID
* ``password`` - the group owner password
If your users are using a SAM pool to log in, they will not have their own platform IDs. The users share a single set of credentials to log into the platform. A SAM pool has a number of IDs to use, but it is not limitless. You may have to account for cases where peak utilization creates a situation where more threads are trying to access the platform than your SAM pool is configured to handle.
The Python SDK is designed to queue during ``Session`` object construction when a SAM pool is fully utilized. During this queuing time the thread trying to construct a new ``Session`` object retries the request every ten seconds.
To ensure that your SAM pool is fully utilized, clean up sessions as soon as your thread is done to prevent a full pool of IDs where all of the threads of database concurrency are in use at the same time. For more information, see :ref:`Cleaning up a session`.
In this example, two threads construct two separate ``Session`` objects and log in with a SAM pool. The application retrieves data from two different tables and downloads them into two separate files.
.. code-block::
import py1010
import threading
import sys
gateway = "https://www2.1010data.com/cgi-bin/gw"
group = "[GROUP_ID]"
owner = "[GROUP_OWNER_USERNAME]"
pwd = "[GROUP_OWNER_PASSWORD]"
tablename = "pub.demo.weather.stations"
def runQuery(table, ops, outfilename):
try:
with py1010.Session(gateway, owner, pwd, py1010.POSSESS,
group) as session:
print("Logged in as {}".format(session.username))
query = session.query(table, ops)
query.run()
with open(outfilename, "w") as outfile:
print("Starting to write to " + outfilename)
for r in query.rows:
outfile.write("\t".join([str(x) for x in r]) + "\n")
except py1010.TentenException as e:
print("Error: {}".format(e))
t1=threading.Thread(target=runQuery,
args=("pub.public_data.census.acs.all_acs_processed_tables",
'', "Output1.txt"))
t2=threading.Thread(target=runQuery,
args=("pub.public_data.census.acs.reference.column_names",
'', "Output2.txt"))
t1.start()
t2.start()
t1.join()
t2.join()
.. note:: [GROUP_ID], [GROUP_OWNER_USERNAME], and [GROUP_OWNER_PASSWORD] are placeholders for valid Insights Platform group ID, owner name, and password.
``flushpool`` (Flush a SAM pool)
--------------------------------
:meth:`py1010.flushpool()` flushes the SAM pool. This ensures that user IDs returned from this point on are fresh and not already logged in.
You may want to periodically request your IDs to start new sessions due to memory overconsumption. You may also want to request IDs to update their view of the database if a table or set of tables have been updated. In these cases, you can use ``flushpool``.
``flushpool`` terminates all sessions once they have been released back into the pool. It will not terminate any allocated or busy sessions.
The SAM pool group owner and the admin of the owner's company can call ``flushpool``. This command does not actually log in the requesting ID.
The ``flushpool`` function takes the following parameters:
* The URL of the 1010 gateway (``url``)
* Group owner name (``owner``)
* Group owner password (``password``)
* Group ID (``group``)
* Path to a log file (``logfile``) - The default value is None.
..
* Mode (not sure)
.. code-block::
py1010.flushpool(gateway, owner, password, group, logfile, mode)
``resetpool`` (Reset a SAM pool)
---------------------------------
You can release all of the user IDs in a specific SAM pool using the :meth:`py1010.resetpool` function.
.. note:: This function is intended for development and debugging.
The ``resetpool`` function takes the following parameters:
* The URL of the 1010 gateway (url)
* Group owner name (owner)
* Group owner password (password)
* Group ID (``group``)
* Path to a log file (``logfile``) - The default value is ``None``.
.. code-block::
py1010.resetpool(gateway, owner, password, group, logfile)
This function will disrupt any user who may be working with a SAM pool ID at the time.
``warmpool`` (Warm a SAM pool)
--------------------------------
One way to reduce the amount of time it takes to retrieve results from the 1010data Insights Platform is to "warm" the SAM pool by having pool IDs logged in ahead of time.
An ID from a SAM pool that was warmed is already logged into the Insights Platform. The :meth:`py1010.warmpool()` function logs in available SAM pool user IDs, which can speed the retrieval of results for users. To this end, you can also provide queries to run on the warmed IDs when they log in.
The ``warmpool`` function takes the following parameters:
* The URL of the 1010 gateway (``url``)
* Group owner name (``owner``)
* Group owner password (``password``)
* Group ID (``group``)
* List of ``BaseQuery`` objects - ``BaseQuery`` objects take a table name and 1010data Macro Language query, but are different from ``Query`` objects because they are not associated with a ``Session`` object. The default value is ``None``.
* Path to a log file (``logfile``) - The default value is ``None``.
Example without queries or a log file path and with them:
.. code-block::
py1010.warmpool(gateway, owner, password, group)
.. code-block::
py1010.warmpool(gateway, owner, password, group, queries=None, logfile=None)
You can run ``warmpool`` without logging in.
``warmpool`` returns the number of IDs warmed.
After the query results are retrieved, the used IDs should be released back into the pool of available IDs. For more information, see :ref:`Cleaning up a session`. These released IDs are not re-warmed automatically. When they are used again, the ID will have to be logged in.
The example below uses ``warmpool`` to warm the SAM pool and ``BaseQuery`` to provide a query:
.. code-block::
#!/usr/bin/env python
import py1010
url = "https://www2.1010data.com/prod-latest/gw"
owner = "USERNAME"
group = "GROUP"
password = "PASSWORD"
queries = [
py1010.BaseQuery("pub.demo.weather.stations", ""),
py1010.BaseQuery("pub.demo.weather.hourly90",
'''''')
]
warmed=py1010.warmpool(url, owner, password, group,
queries=queries,
logfile="FOOO.log");
print("Warmed: %d"%warmed)
.. note:: ``USERNAME``, ``GROUP``, and ``PASSWORD`` are placeholders for valid Insights Platform group owner name. group name, and password.