DataBlazer Changelog DataBlazer consists of TenUp, TenDo, and Data Hauler. Tenup Changelog 20221202 - Added AWS S3 support. (CT-260) AWS S3 can be used as a data source when uploading a table, using the spec file: filename.csv Note that api_key is the name of a key that must be added to your 1010data uid before attempting the upload (use --add-key) 20221028 - Fixed command line help formatting and added environment variables section. (CT-252) 20221025 - Added Azure Blob Storage support. (CT-219, CT-226) - Key management options allow you to add or remove a cloud storage key from your 1010data uid settings: $ --add-key="name;value" --key-type=abs $ --rm-key=name --key-type=abs Note that --add-key value must be quoted so that the semicolon delimiter is not interpreted by the shell. - Azure Blob Storage can be used as a data source when uploading a table, using the spec file: filename.csv Note that the api_key is the name of a key that must be added to your 1010data uid before attempting the upload. 20220921 - Added M1 Mac build of tenup for download (CT-145) NOTE: only works for Mac OS 12.3 or higher 20220810 - CSDK/SSO update 20220725 - Fixed bug in --ftp-upload, which caused a "Failed to find files in path" error when local filenames are specified using an absolute path. (CT-217) 20220627 - CSDK/SSO update 20220614 - Fix to sso_login to use POST methods when needed. 20220525 - THE STRIPEFACTOR OPTION IS NOW DEPRECATED. (STRIPEFACTOR is specified in the spec tree in a tag under the tag, or on the command line using --stripefactor). We will continue to support it for some time for api=addtab (flat file) uploads but all users should update their commands to use the new STRIPE option described below. - The new STRIPE option indicates the number of the user's machines to replicate the data on. STRIPE is an integer specified in the table tree in a attribute 'stripe', or on the command line using --stripe:
--stripe=4 This results in table data on 4 machines. STRIPE is supported for api=chunkload (ODBC) and api=addtab (flat file) uploads. You might notice that STRIPE means the same thing as the deprecated STRIPEFACTOR option for api=addtab (flat file) uploads. We ask all users to switch to STRIPE instead. - The new STRIPE_FACTOR option indicates the fraction of the user's machines to replicate the data on. STRIPE_FACTOR is a decimal between 0 and 1 specified in the table tree in a
attribute 'stripe_factor', or on the command line using --stripe_factor:
--stripe_factor=0.5 This results in table data on 0.5 of the user's total machines (If user has 4 machines, data on 2). STRIPE_FACTOR is supported for api=chunkload (ODBC) and api=addtab (flat file) uploads. STRIPE_FACTOR is unrelated to the now deprecated STRIPEFACTOR. 20211207 - Improved tenup logging so that ODBC driver warnings are always printed to stdout (CORE-5855). 20210618 - Added support for a new SSO authentication tag (CORE-5477). 20210318 - Added support for a new SSO authentication tag (CORE-5362). 20210305 - Fixed precision issue in data downloaded from datetime columns (CORE-5314). 20210225 - Add limited SAML/SSO login support with the --authentication tag. The datablazer package includes the sso_login executable that must be on the PATH to use this feature. Windows users that run the installer do not need to manually update their PATH. (CORE-5362) 20201006 - (mac) rebuilt to link to dependencies in /usr/lib rather than /opt/local/lib... again. 20200812 - (windows) installation package 3.00.0000 removed outdated tenup_guide.pdf from installation, and associated shortcut. - updated download link printed to STDOUT when there is a new version of tenup available. 20200806 - removed the default MAXDOWN (4194304). tenup no longer uploads a table with a MAXDOWN unless specified --max-down or <table.maxdown> (in the table tree). refer to command line usage for a description of MAXDOWN. 20200803 - (mac) rebuilt to link to dependencies in /usr/lib rather than /opt/local/lib. 20200706 - updated digital signature certificate (windows) 20200527 - Fixed bug in checking the latest deployed version of tenup, which caused 404 Not Found error. 20200115 - Updates for login through OAuth2 single sign-on (SSO). This type of login is indicated by providing your OAuth2 SSO email address as UID (-u/--username/TENTENUID). PWD (-p/--password/TENTENPW) is then interpreted as your OAuth2 SSO account password, and GW (-g/--gateway/TENTENGW) as an optional version specifier (e.g. beta-14.44). Users with multiple 1010data environments associated with their OAuth2 SSO account must provide ENVID (--env-id) or ENVDOMAAINNAME (--env-domain) to target a specific environment. You can run tenup without either of these options to generate a list of valid ENVIDs and ENVDOMAINNAMEs that you can reference in your subsequent commands. 20191010 - Recompiled Linux (lin64) build to link against older version of libc in order to run on supported Linux distros. 20190913 - Updated build to prevent symbol conflict between the libraries linked into tenup for Linux and different versions of those libraries that might be loaded by ODBC drivers (--connection-string=CONNSTR). 20190903 - Adding Automatic Proxy Configuration support using the new proxyconfig Library, Auto proxy is supported on all platform. Using this option is similar to tendo: -P PROXY for using manual proxy -P.PACFILE for using PAC file to get the proxies from -P. for using auto proxy detection. - Extended support for upload timestamp columns to flat file uploads. Upload timestamp column names and labels can be indicated in the the command line options --upload-date and --upload-datetime and <table> attributes userdatename, userdatelabel, userdatetimename, and userdatetimelabel. - (Windows only) Command line input will now be pre-processed into its UTF-8 representation. This enables table and column name autoclean of the positional argument NEWTABLE and optional argument -h/--column-names=NAME1,NAME2,...,NAMEn for users running tenup with a system Windows/ANSI code page other than 65001 (UTF-8). The current system Windows/ANSI code page determines the encoding of command line arguments that tenup receives. - The --report=REPORT option has been extended to flat file tenup. The generated files REPORT.table and REPORT.spec are distinctly useful as
and input to tenup. REPORT.table contains the exact upload
(the transaction log generated by --xmllog also contains the same
). REPORT.spec contains the upload modified with local filenames under . This is done to enable re-use with tenup. - If REPORT contains environment variables, the filename logged in the options parsed summary has changed to the result of environment variable expansion. The options parsed summary is logged to OUT at verbosity level 2 (-v/--verbose used at least twice). - The upload path of the table can be indicated with the option --upload-path=UPLOADPATH when it is not indicated in the positional argument NEWTABLE. This option introduces command line usage to override the upload path indicated in the table tree (
attribute). - Removed using column display width reported for text columns when determining column size. 20190529 - The --ftp-upload boolean flag is available to indicate that the flat file(s) should be uploaded to 1010data over SFTP and the table should be created with non-transactional addtab. This feature requires a user UID with password PWD on the FTP server associated with the environment referenced in GW. Transactional addtab (default for flat file uploads) is useful for detecting errors in the data or spec as they are encountered. Non-transactional addtab can be used to improve performance of established uploads that are known not to fail. 20190117 - The --ignore-null/ option is available to indicate that each null character ('\0') in the flat file input data should be uploaded as a space character (' '). - The --stripefactor/ option indicates how many replicas of each segment of the flat file input data to store. - The --limit-rows/ option indicates how many leading rows of the data to upload (the remainder are skipped). - The --nowrite-columns/ option indicates the columns in the flat file input data that should be read but not uploaded. Unlike --skip-columns/, these columns remain available in the input expressions (--column-expressions/) of the other columns. - The tag is recognized as an alias of to indicate the name of a column in flat file input data. - Fixed a bug parsing multiple tags. - Updated command line usage to note that link headers are deprecated after version prime-12.53. 20180919 - Fixed issues with --pipeline on Windows and Linux. On Windows, using this option could have crashed prior to this update. On Linux, the issue would have resulted in the upload failing with the error "Reponse from server had wrong columns or number of rows" - Improved --help usage text. - The "basename" of a table (the last token in the dot-separated path NEWTABLE) is subject to autocleaning prior to upload. Autoclean will normalize non-ASCII (UTF-8 multibyte) characters where possible, prepend 'x_' to identifiers leading with digits or underscores, and replace inner whitespace with underscores. Normalization is limited to a subset of Unicode characters on the basic and supplementary multilingual planes for which standard decomposition is defined. Autoclean does not remove or replace any non-whitespace ASCII characters. Note: The default table title is determined by the table name *before* cleaning. For example, if you specify the name "foo._bar" and omit a title, the resultant table is named "foo.x_bar" and titled "foo._bar". Note: Table names are validated against the 1010data character set and checked for duplication *after* cleaning. The user is responsible for table names that cannot be normalized and for conflicts with existing table names as a result of cleaning. - Column name autoclean has been extended to prepend 'x_' to identifiers leading with underscores (e.g. '_foo' is cleaned to 'x__foo'). Note: The default column labels (titles) are determined by column names *before* cleaning. For example, if you specify the name "_foo" and omit the title, the resultant column is named "x_foo" and titled "_foo". Note: Column names are validated against the 1010data character set and checked for duplication *after* cleaning. The user is responsible for column names that become duplicated as a result of cleaning. 20180321 - Fixed bug in CSDK (tenten_PrepareUpload) causing undefined behavior. 20180319 - Fixed bug causing "tenten_UploadBind???Ptr - 1010data Error - -16386" errors. 20180315 - --auto-correct enables a two-fold repair of erroneous input during flat file upload. This process is limited to errors caused by unpaired masking characters and/or an unequal number of delimiters (column separators) across records. A first pass over the input tracks the total number of masking characters encountered. A masking character is assumed unpaired and is inserted at each EOR that this number is odd. A second pass over the input tracks the number of delimiters encountered per record. A record is either truncated or padded (with NULL) at each EOR that this number deviates from the expected number of columns. --auto-correct is not suitable for use with data expected to contain EOR character(s) within masked fields. - Fixed a bug in the procedure that determines a gateway URL when -g/--gateway is omitted and the environment variable TENTENGW is non-empty and does not define a fully qualified URL. - If a gateway GW (specified via -g/--gateway/TENTENGW) is not a fully qualified URL nor a valid shortcut (www2, nj, www2test, njtest, nyx, ny, ra), it is assumed a CGI script directory and resolves to "https://www2.1010data.com/cgi-bin/[GW]/gw.k". - --auto-spec indicates that input metadata should be detected from flat files and used to populate metadata omitted by the user. This process is analogous to metadata extrapolation from an ODBC data source. The following subset of metadata is considered: record type, column separator, record delimiter, column names, column labels, column types, input column types, and column formats. Yield is conditional on the amount/accuracy of metadata described by and detected from flat file input. --auto-spec cannot be used on input that is fixed-width or parameterized by a non-default mask width and/or character. - Accepted
attributes have been extended to include the boolean "responsible" attribute, a counterpart to the existing long option --responsible. - Column separators can now be specified by name, which allows use of characters that are hard to input on the command line. For example, to load a flat file with tab-separated columns, use the keyword "tab" as an argument to the long option --column-separator or as the content of the tag. - Tenup now accepts XML files as input for flat file uploads. The filename can be specified as a standalone unswitched argument or as the second unswitched argument after the
(table tree XML) filename. In either case, the filename must be prefixed with '@'. Tenup supports a subset of the tags documented at http://www.1010data.com/downloads/apimanual.pdf (see "addtab" under "Powerloader"). Valid tags on XML level 1 are source, rectype, sep, eor, maskw, mchr, begrecs, name, title, ldesc, sdesc, link, arch, ts, cols, and users. Valid tags on XML level 2 are file, user, and col. Valid tags on XML level 3 are name, head, help, format, fix, index, skip, case, order, bord, type, width, and exp. Valid tags on XML level 4 are type, width, and dec. - --index-columns and are available in addition to the existing -X/--column-indices and already supported "index" attribute of to specify columns that will be indexed in the uploaded table. The expected argument to -X/--column-indices has changed from a list of positional boolean flags to a list of column names. The long option --index-columns is an alias for -X/--column-indices. - -F/--fix-columns and are available in addition to the already supported "fixed" attribute of to specify columns that will be fixed (leftmost and non-scrolling) in the uploaded table. - --skip-columns and are available to specify columns that will not be read from the flat file and thus will not be written to the uploaded table. - --lowercase-columns and are available to specify text columns in the flat file that will be forced into lowercase prior to upload. - --uppercase-columns and are available to specify text columns in the flat file that will be forced into uppercase prior to upload. - --column-orders and are available to upload columns in an order that differs from the order they appeared in the flat file. - --fixed-width and are available to upload fixed-width columns from a flat file. A non-zero field width must be provided for each column using --input-widths or . - --big-endian, --little-endian, and are available to specify the byte storage order of an input flat file. Because little endian is assumed by default (if none of these options are provided), the primary purpose of --little-endian is to override any byte storage order set in an input . - --input-types and can be used to specify the types of the columns in a flat file. This differs from -i/--column-types, which refers to the types of columns in the uploaded table. If either -i/--column-types or --input-types are omitted during a flat file upload, the other is used as default. - --input-widths and can be used to specify the field width of the columns in a flat file. Input widths can be provided for both separated and fixed-width columns. If columns are separated, any number of input widths less than or equal to the number of columns can be specified. If columns are fixed-width, the number of input widths specified must be equal to the number of columns. - --input-expressions and can be used to define "input" (value) expressions for computed columns. An input expression may only refer to the preceding columns that have not been "skipped" (using --skip-columns or ). - Additional timestamp columns containing the datetime or date of table upload can be included in any upload from an ODBC data source. To indicate a timestamp column, the user must provide a column name and optionally a column label. Command line options --upload-datetime and --upload-date, as well as
attributes "userdatetimename", "userdatetimelabel", "userdatename", and "userdatelabel", are available to do so. Timestamp column values are computed internally and cannot be parameterized by the user. 20170925 - The --mask-character option can be used to override the default masking character (double quote) of the flat file being loaded. The command argument must represent a printable (non-control) ASCII character, with the exception of 'none' and '\0' used to represent the empty string. Note: A masking character is the character that encapsulates variable width text fields in a flat file. The masking character should not be embedded within any single text field. It should not be set to 'none' if the column separator is embedded within any single text field. In either of these cases, the results of parsing the flat file are not well defined. - Column name autoclean has been extended to normalize non-ASCII (UTF-8 multibyte) characters where possible and prepend 'x_' to identifiers with leading digits. Normalization is limited to a subset of Unicode characters on the basic and supplementary multilingual planes for which standard decomposition is defined. Autoclean does not remove or replace any non-whitespace ASCII characters. Note: The default column labels (titles) are determined by column names *before* cleaning. For example, if you specify the name "_foo" and omit the title, the resultant column is named "x_foo" and titled "_foo". Note: Column names are validated against the 1010data character set and checked for duplication *after* cleaning. The user is responsible for column names that cannot be normalized and for column names that become duplicated as a result of cleaning. 20170901 - The command line option -i/--column-types and the 'type' attribute of the table tree tag
now take an argument 'j' to load a column as a 64-bit integer column. - When using -i/--column-types, any type omitted for a SQL_BIGINT column defaults to 64-bit integer. To load as double precision floating point instead (prior default behavior), please specify the 'f' type. - When using -i/--column-types, any type provided for a SQL_BIGINT column is prioritized over --use-long-ints. For example, a SQL_BIGINT column will be loaded as 32-bit integer if the 'i' type is specified. To load as 64-bit integer instead (prior behavior with --use-long-ints on), please specify the 'j' type (or no type). - A --use-doubles option has been added to load SQL_BIGINT columns as double precision floating point columns. This is currently the default behavior, unless using -i/--column-types. -i/--column-types may not be used with --use-doubles. - In the future, SQL_BIGINT columns will be loaded as 64-bit integer columns by default. --use-doubles may be used at that time to revert to a double default. - The -m/--max-column-size option can be used to override the default mask width (5000 characters) when uploading data from a flat file. - The --skip-rows option can be used to omit some number of leading rows when uploading data from a flat file. 20170503 - Tenup now checks for updated versions on the new website where client side tools are hosted. 20170315 - Added support for uploading 64-bit long intger columns. - Added the --default-column-width switch, to control the width attribute in the 1010data format string generated for columns where a width is not explicitly provided by the user. - Tenup now will expand paths in cases where your shell doesn't. - Fixed an issue where an unhelpful error was presented to the user instead of the error provided by 1010data. - Tenup now checks for updated versions on the new website where client side tools are hosted. 20170212 - Added support for appending to a table when uploading a flat file. 20170130 - Fixed issue with automatically generatec specifications for flat files that caused some integer columns to load as floats. - Tenup now properly supports "i"/"f"/"a" as as the value of the "type" attribute of the tag in the tree. 20161222 - Added support for flat file uploads. 20161207 - Improved support for HTTP redirects (necessary in an HA 1010data environment). 20160929 - Changed how URL redirects are handled internally to reduce network traffic. - Updated certificate authority bundle used by libcurl. 20160427 - Improved error logging. - Fixed an issue that caused tenup to crash just before exit when using --idle-poll. - Changed the default -G/--log-interval to 2^23 (previously 2^27). - Changed the default text column width to 12 characters (previously 30). - Added --prompt-proxy-pwd allows a user to avoid putting their proxy password in plain text. 20150923 - Improved error messages. 20150901 - Added -~/--idle-poll, which causes tenup to poll 1010data in the background while waiting for the origin database to execute a query or return results. This is useful if the origin database takes longer than your UID's dormant time to execute a query or get data. - Added --no-empty=EMPTYEXIT, which causes tenup to exit with program exit code EMPTYEXIT if it would have otherwise created an empty table in 1010data. - Added -:/--param, which allows the user to specify parameters to their ODBC query. 20150806 - Added OpenPGP signature. - Improved retry logic. 20150715 - Fixed URL truncation issue for uploads with many columns. 20150708 - Improved retry logic. - Documentation updates. 20150430 - Improved connection stabilization logic. 20150311 - Resolved an issue that prevented tenup from loading data. - Fixed path updating bug in Windows installer. 20150309 - Optimized backoff during high utilization of a UID or SAM pool. - Improved logging. 20141219 - The "owner" attribute of the
tag is now supported in spec files. - Added the --pool-retry option, which makes the number of requests for a UID in a SAM pool configurable (the default of unlimited attempts has not changed). - Improved logging. - Failures from the system functions time and localtime are now tolerated. 20140906 - When logging into a pool session, any previous chunkload data is aborted before starting the current upload. 20140822 - Added the -//--restarted switch. In the event of a client side error, tenup may be restarted with -//--restarted to attempt to pick up where it left off. When doing this you must submit the *exact* same arguments. The only exceptions are you need to add -//--restarted and you must use -K/--possess or -s/--sid-epw. If the results of your query may have changed between the first and second run -//--restarted is not recommended. - Fixed a bug which caused -2/--pipeline to always fail on Windows. - Improved error messages. - Miscellaneous bug fixes. 20140813 - Improved logging. 20140811 - Fixed a deadlock when using -2/--pipeline and loading tables with more than BUNDLE columns. - Fixed a random failure when using -2/--pipeline on Linux. - Fixed a race condition when using -2/--pipeline that occurred when the total amount of data loaded modulo WINDOW was equal to zero. - Improved logging. - -8/--utf-8 and -w/--wide-only can no longer ber used together. - When tenup encounters an error it now consistently terminates with a non-zero program exit code. 20140805 - When using the --log-data switch SQL_W[[LONG]VAR]CHAR columns will have their pre-converted raw hex values written to the log. - Fixed a crash when parsing XML spec files. - Fixed a bug in -2/--pipeline. 20140731 - Added the -2/--pipeline switch which allows the user to request that tenup pull data from the source while it transmits data to 1010data. - More consistently converts column names to lower case. - Improved error messages. 20140718 - Fixed a bug which prevented tenup from finalizing loads. - Fixed a bug which prevented the use of the -+/--time-series switch. - Fixed an error in the table tree when -+/--time-series was used. - Tenup now allocates slightly more space for text columns. 20140716 - -0/--owner added to allow the user to set a different user/group as the owner of a table after upload. - Fixed the --sort option. - Minor updates to the tenup User Guide. - Fixed an issue related to SQL_TYPE_TIME columns that were reported to have precision == 0. - Tenup now supports columns that contain "internal" spaces (leading and trailing spaces still throw errors) - Previously -j/--segby enforced sortseg and -J/--sortseg enforced segby, this has been corrected. - When checking for a new version of tenup HTTPS is now used. - Added AIX support (binary not included in standard package). - Added automatic table name generation based on UID/time/PID. - Connection strings containing the key 'pw' or 'password' (case insensitive) will now have their values masked out. - -+/--time-series has been overhauled. - When using a SAM pool rc=38 "system is busy" errors now result in additional attempts to acquire a free session. - When using a SID,EPW or possessing a session tenup will clear your cache before it begins to chunkload data. - Improved some of tenup's error messages. - Fixed a bug when -W/--window was less than -R/--fetch-size. 20140605 - -./--base-table added to allow the user to specify a table where column remarks can be sourced and used as column descriptions. - When -B/--bug-report is used without explicitly calling -o/--out, -O/--err and/or -x/--xmllog the file names used for the output log, error log and xml log are now based on the name of the bug-report. Additionally, those files are deleted from disk after tenup finishes execution (they are still included in the .1010zip) - Column names provided by the user or the database are now validated as legal 1010data columns names. - Tenup now sends the destination table metadata to 1010data before the first call to SQLFetch. This will cause inevitable errors (insufficient permission to create the table) to appear earlier in the load process. - When a user manually specifies column types/formats in a spec file or on the command line tenup will now much more aggressively validate that the types/formats specified are legal conversions from the columns' ODBC SQL types. - The -+/--time-series switch is no longer ignored. - Improved logging. 20140527 - -z/--bundle-size added to allow the user to configure how many columns are included in each message sent to 1010data. - When using SAM pools -l/--logout is no longer required to release your session before exit. - Improved logging. - Improved performance. - Miscellaneous bug fixes. 20140515 - When tenup encounters an error validating/converting text data it will now include the problem column's name and the first row where the error occurred in the error message. 20140514 - A complete user guide is now included in tenup.zip. - Tenup's XML log now supports long URLs. - Added Windows example batch script. - Minor corrections to internal documentation. - Miscellaneous bug fixes. 20140509 - Tenup can now load integer columns that already have their data in 1010data style YYYYMMDD format. - Improved logging. 20140422 - Added OSX support. 20140418 - Update from OpenSSL 1.0.1e to 1.0.1g. 20140403 - When upload a table, tenup will now send a windows of rows as a single POST, instead of one post per column. - Miscellaneous bug fixes. 20140314 - -8/--utf-8 added for source databases that return properly encoded UTF-8 data in SQL_[[LONG]VAR]CHAR columns, that don't need to (or can't) be returned as SQL_W[[LONG]VAR]CHAR columns. - Previously, if the PWD key was included in the connection string argument (-C/--connection-string) there the unmasked password appeared in the log. 20140307 - -B/--bug-report now takes an argument, which is interpreted as the file where an archive of OUT, ERR and LOG will be stored. 20140306 - When '`' is included in a column header it will now be automatically converted into a '\n', conforming with other upload mechanisms. 20140303 - Previously, when uploading SQL_W[[LONG]VAR]CHAR columns, if -R/--fetch-size was N then the (X*N)+[N/2,N] for all values for X (the second half of each window of data) resulted in NULL data. This error has been resolved. - Tenup now converts the data in SQL_W[[LONG]VAR]CHAR columns to UTF-8 more efficiently. - When tenup asks an ODBC data source to return N rows at a time (as specified by -R/--fetch-size) and the ODBC data source responds that it will return a different number of rows at a time, tenup will now allocate (presumably) smaller internal buffers to hold the data, saving memory. - When loading SQL_W[[LONG]VAR]CHAR columns, if your data is too wide and you get a "01004" error (data right truncated), tenup will now more accurately suggest a value for -m/--max-column-size. 20140228 - Tenup will now throw an error if it detects data outside of the 7-bit ASCII range in SQL_[[LONG]VAR]CHAR columns. - Added -w/--wide-only option, which forces tenup to request SQL_C_WCHAR data from the ODBC data source when it would have normally request SQL_C_CHAR data. This is necessary if SQL_[[LONG]VAR]CHAR columns contain data outside of the 7-bit ASCII range. 20140226 - Corrected a memory access violation. - Links specififed in a spec file that are not self closing are now handled correctly. - If in the spec file, or on the command line a column is given type="i" and a date format or if a column is given type="f" and a timestamp format tenup will now properly handle the conversion. - If no prelinks/denorm directives are given Tenup will totally omit the tag (this was causing a rank error in some versions). - If an ODBC 01004 error (data truncation) occurs, tenup will now report how wide the text buffers need to be to accomodate the data. - Corrected errors in internal usage documentation and improved some error messages. 20140220 - Improved performance when buffering/preparing/compressing data for upload. - Improved default behavior when loading SQL_[TYPE_]TIME columns that do not have fractional second precision. - The -a/--append switches now work, allowing a user to append to an existing table. - The -G/--log-interval switches have been added, allowing a user with -vvv or less to get log information intermittently while loading data (-vvvv already logs during the data loading process, but may not be desireable as logs can get large). - The -r/--report option will now write a (tenup usable) spec file to disk. - 32 and 64 bit Windows application installers are now included in tenup.zip. - Miscellaneous bug fixes. 20140131 - The following transformations (applied after load) are now supported: resegmentation (segby/sortseg + advises), prelinking, denormalizing, sorting (within a segment) and indexing. - Table tree XML specification files are now fully supported. - When loading string data performance has been dramatically improved if the -R/--fetch-size buffer was substantially smaller than the -W/--window buffer (if the origin ODBC data source could only return one row at a time performance should also be improved). - Adjusted the default maximum string column width, which should improve the default memory footprint when loading string data. - The -T/--ignore-trunc switches now behave properly (in general more care is taken to raise an error if string data is truncated). - SAM pools are now supported. - Miscellaneous bug fixes. 20131216 - When verbose level is set to 3 (-vvv) the output log will no longer contain entries for each call to SQLFetch/tenten_UploadRows. 20131121 - Added ability to log raw data, for debugging purposes. - Several typo corrections in internal documentation/error messages. 20131114 - You can now overload a column's type and/or format. - You can now specify a column's description field. - Tenup will no longer pop up an ODBC driver's configuration panel when an error occurs during SQLDriverConnect. - When using the -S/--out-sid-epw switch the timestamp will longer be printed. - Added option to abort old chunkload sessions before starting new ones. - Added a maximum decimal precision (9) to avoid chunkload errors. - Minor corrections in internal documentation. - Minor corrections in log output. - Miscellaneous bug fixes. 20130625 - Fixed several bugs which would cause tenup to crash. - Fixed the -T/--ignore-trunc switch. - Fixed the -y/--replace switch. - Extended -r/--report output. - Extended internal help. - Improved memory usage. - Improved password masking in logs. - Improved configuration option logging. - Changed the -x/--xmllog switch to make sure you get an xml log, even if you get an error during login to 1010data. 20130529 - Private beta deployment. - This release (and all following releases unless stated otherwise) is being developed on Ubuntu 12.04.5 (x64). The linux distribution is compiled/deployed on Ubuntu 7.04 targeting GLIBC 2.5 with gcc 4.2.4. We fully expect it to work on other linux distributions and versions, so long as they have GLIBC 2.5 or newer, and conform to the C++03 ABI. The Windows release is compiled and tested on Windows 7, but we again expect it to work on other versions of Windows. Tendo Changelog 20230320 - New --save-compression option, which accepts values "none", "zip", or "gzip". Use with ftp: or cloud: syntax, which saves results of query to FTP or cloud storage location. (CT-286) 20221202 - Added AWS S3 support. (CT-260) Query results can be uploaded as files to AWS S3: $ cloud:filename.txt=SQL_QUERY --key-type=s3 --key-name=name \ --key-bucket=bucket or $ cloud:filename.txt=TABLE QUERY --key-type=s3 --key-name=name \ --key-bucket=bucket Note that --key-name is the name of a key that must be added to your 1010data uid before attempting the upload (use --add-key). 20221028 - Fixed command line help formatting. (CT-252) 20221025 - Added Azure Blob Storage support. (CT-220, CT-225) - Key management options allow you to add or remove a cloud storage key from your 1010data uid settings: $ --add-key="name;value" --key-type=abs $ --rm-key=name --key-type=abs Note that --add-key value must be quoted so that the semicolon delimiter is not interpreted by the shell. - Query results can be uploaded as files to Azure Blob Storage: $ cloud:filename.txt=SQL_QUERY --key-type=abs --key-name=name \ --key-account=account --key-container=container or $ cloud:filename.txt=TABLE QUERY --key-type=abs --key-name=name \ --key-account=account --key-container=container Note that --key-name is the name of a key that must be added to your 1010data uid before attempting the upload. 20220921 - Added M1 Mac build of tendo for download (CT-145) NOTE: only works for Mac OS 12.3 or higher 20220810 - CSDK/SSO update 20220627 - CSDK/SSO update 20220614 - Fix to sso_login to use POST methods when needed. 20220525 - Added --sam-status 20210618 - Added support for a new SSO authentication tag (CORE-5477). - Fixed bug so that content is wrapped in CDATA when it contains whitespace, when saving to FTP (CORE-5629). 20210318 - Added support for a new SSO authentication tag (CORE-5362). 20210305 - Fixed precision issue in data downloaded from datetime columns (CORE-5314). 20210225 - Add limited SAML/SSO login support with the --authentication tag. The datablazer package includes the sso_login executable that must be on the PATH to use this feature. Windows users that run the installer do not need to manually update their PATH. (CORE-5362) 20201020 - --owner is now respected for saving queries with --query (CORE-5143). 20201006 - fixed bug introduced in v20200803 to prevent submitting api=reluid transaction after sessionless transactions like api=resetpool (CORE-5117). 20200831 - (linux) rebuilt against an older version of glibc (2.10). 20200812 - (windows) installation package 3.00.0000 removed outdated tendo_guide.pdf from installation, and associated shortcut. - updated download link printed to STDERR when there is a new version of tendo available. 20200803 - (mac) rebuilt to link to dependencies in /usr/lib rather than /opt/local/lib. 20200706 - updated digital signature certificate (windows) 20200527 - Fixed bug in checking the latest deployed version of tendo, which caused 404 Not Found error. 20200115 - Updates for login through OAuth2 single sign-on (SSO). This type of login is indicated by providing your OAuth2 SSO email address as UID (-u/TENTENUID). PWD (-p/TENTENPW) is then interpreted as your OAuth2 SSO account password, and GW (-g/TENTENGW) as an optional version specifier (e.g. beta-14.44). Users with multiple 1010data environments associated with their OAuth2 SSO account must provide ENVID (--env-id) or ENVDOMAAINNAME (--env-domain) to target a specific environment. You can run tendo without either of these options to generate a list of valid ENVIDs and ENVDOMAINNAMEs that you can reference in your subsequent commands. 20191031 - Added --link=TABLE:LOCALCOLS:FOREIGNCOLS[:DENORMCOLS] option for prelinks support. - Added --segment-size=SIZE option to specify the size of segments to to be made. - Added --table-tree=FILE option to dump a table's metadata in XML format. - Rendering files with --data=., --xlsx=., --pdf-qa=., --static-html=. will use the filename supplied by the server side, and even if a name is specified, append the appropriate extension, unless it is redundant. - Added missing docstring for --debug-accum option 20190905 - Changing the Automatic Proxy Configuration library to use the new one, Now it is supported an all platforms. Using this option is the same it was, and now it is documented in the help. 20190816 - Fixed bug when using --get=FILE and --pdf=FILE. 20190529 - Initial build for the new DataBlazer suite installer. 20170723 - Fixed a bug where option --data=FILE could fail for small files. 20170526 - Fixed an incompatability with versions of 1010data older than 9.18. 20170518 - Added option --resetpool to manage the state of an id pool. 20170510 - Added option --static-html=FILE, which will render the query into a static html page and save that output into FILE. 20170503 - Tendo now checks for updated versions on the new website where client side tools are hosted. - Fixed an issue on Windows where very long queries were unceremoniously truncated. - New options: --flushpool and --warmpool to manage the state of an id pool. 20161207 - Improved support for HTTP redirects (necessary in an HA 1010data environment). 20160929 - Changed how URL redirects are handled internally to reduce network traffic. - Updated certificate authority bundle used by libcurl. - Fixed an edge case when writing columns with very wide strings that contain CSV quote characters. 20160823 - Added option -M to report the number of columns in a result set without downloading the data. If used with -N, # of columns is reported before the row count. - Changed how URL redirects are handled internally to reduce network traffic. 20160205 - Fixed bug when specifying an FTP file path with spaces in it. Example: tendo [...] "ftp:my results=base.table" [...] 20151210 - Added the --freshen switch, which causes tendo to update your session's view of the tables your query depends on before executing the query. - Fixed bug in --uploaders=IDLIST that was preventing the options from correctly applying with --mkdir=DIR. 20151001 - Added the --data switch, which downloads a result set though the QuickApp data renderer. 20150811 - Added --owner=ID option to specify the owner of a table or directory when saving a atable, merging tables, making a directory, or modifying table or directory metadata. 20150806 - Added OpenPGP signature. - Fixed a bug when mergeing tables - Improved retry logic. 20150708 - Improved retry logic. - Documentation updates. 20150617 - Added documentation for the advanced savetable features that were added in 20150309. 20150427 - Added support for new "type:currency" column format type when outputting formatted data (-f/-F). - Fixed issue when writing string data with length greater than 16KB. 20150403 - Added the --pdf-qa switch, which downloads a result set though the QuickApp PDF renderer. 20150309 - Optimized backoff during high utilization of a UID or SAM pool. - Added --mkdir option. - Added support for advanced savetable features (--segby, --sortseg, --segby-advise, --sortseg-advise, --time-series, --sort, and --time-series). 20150112 - Updated documentation. - Added support for legacy encryption systems. 20141124 - Minor update to check latest version over https. 20141107 - Added the --query switch, which allow tendo to create quick queries on the 1010data platform. 20140806 - Fixed the handling of type mismatches between a reported column type and the actual downloaded data type when --lenient used. - Added the --update switch, which refreshes your session's view of a table or directory before the query is run. 20140429 - Fixed issues with the -F option: fixed-width output. 20140418 - Update from OpenSSL 1.0.1e to 1.0.1g. 20140409 - Added output formatting for "ansidate" and "ansidatetime" column format types with -f option. 20140402 - Revised usage/help output. - Updated the tendo Windows installer to fix issues when appending to a long PATH environment variable. Will prompt when it cannot add the install directory to PATH (2047 chars). 20140331 - Added --xls=FILE.xls switch, which downloads a result set as an excel compatible .xls file. - Added --pdf=FILE.pdf switch, which downloads a result set as a paginated .pdf file. 20140218 - When www.1010data.com isn't resolvable tendo will no longer make multiple attempts to check if a new version is available. 20140128 - Added support for new compression types. 20131209 - When saving/appending to a table if you do not have permission to overwrite the table in question you will now get an error before running the query. - Result sets with more than 1000 columns are now supported. 20131118 - Fixed issue introduced in 20130805 which caused --xmllog=foo.1010xml to be ignored when logging in with -s/TENTENSID. 20131023 - Tendo will now read all entries in a substitution file, even if the file does not end with an empty line. 20131017 - Fixed issue with the Windows installer not creating the uninstaller. 20130819 - When connected to 1010data via a proxy and saving/updating a table you will now benefit from the same retry logic that users who aren't using a proxy enjoy. - Tendo will no longer make multiple attempts to determine the most recent version at startup. - Updated how directory information is parsed to support newest API updates. 20130805 - Fixed issue with XML logging and SAM pools. 20130520 - RHEL5 supprt restored (broken in 20130516). 20130516 - A large update to the tendo user guide is included in this release. - Added the -L option, which explicitly tells tendo not to logout at the end of the session. 20130408 - Resolved the issue where if you possessed a session using the -s switch you would be unable to get forced retry via -~ for savetable and savetoftp transactions. 20130403 - Added option -K to possess a session during login. If the user has a 1010 session active, tendo will use that session for the query. If there is no active session, a new one is started. When -K is used, tendo will not perform a logout after the query by default (unless -l is also used). This allows for a series of tendo -K runs to utilize the same session. This works similarly to the -S/-s option without needing to record the the specific session credentials. 20130402 - Corrected an issue which was preventing http keep-alive packets from being sent. - This release (and all following releases unless stated otherwise) is being developed on Ubuntu 12.04.5 (x64). The linux distribution is compiled/deployed on Ubuntu 7.04 targeting GLIBC 2.5 with gcc 4.2.4. We fully expect it to work on other linux distributions and versions, so long as they have GLIBC 2.5 or newer, and conform to the C++03 ABI. The Windows release is compiled and tested on Windows 7, but we again expect it to work on other versions of Windows. 20130320 - Tendo now installs into the same folder as the ODBC driver. 20130118 - Previously during data download if what was supposed to be valid compressed data or an XML error code was anything else tendo would crash, this has been resolved. - Improved XML logging. 20121231 - Improved verbose output. 20121220 - Improved the connection stability logic in transactions that download data. 20121210 - Added logging of tendo version and system version to verbose logging. 20121024 - Previously, if you could connect to www.1010data.com but after connection the server failed to serve up the tendo version file promptly tendo could hang. This has been corrected. 20120904 - Removed several command line switches from the output of "$ tendo --" that were not implemented. 20120822 - Removed a leftover debugging statement output when the -f option was used. - Added --quoteall, which wraps all CSV fields in the CSV quote character, even if the field doesn't contain a quote character, row seperator or column seperator. 20120816 - When using the --xmllog=someFile.txt option passwords are now masked out of someFile.txt. - When using the --xmllog=someFile.txt option with the --pool=samPoolName option the getuid transaction will now be included in someFile.txt. - When using the --xmllog=someFile.txt option and an error occurs during login someFIle.txt will now be generated (before you only got someFile.txt if you successfully got past login). - Corrections to tendo's internal help info ($ tendo --). 20120814 - RHEL5 supprt restored (broken in 20120813). 20120813 - Corrected an issue when printing formatted floating point columns. 20120601 - The logic to check if tendo was up to date was causing the applicatino to hang if "www.1010data.com" was not resolving due to firewall/etc. Tendo will now spend a maximum of 5 seconds checking if an update is available. 20120524 - Changed several static buffers to dynamic buffers to better support some newly released features. - Removed references to an invalid option from the documentation. 20120516 - When specifying a new table's long description if LDESC is "*" then the query is substituted (similar behavior to the GUI). 20120502 - When --xmllog=FILENAME.log is specified, if tendo fails to open FILENAME.log a warning will be emitted and an attempt will be made to log to the default log location (./1010csdk.log). 20120416 - Previously savetable/mergetable/etc. were not automatically retryable in the event of a network timeout, this is no longer the case. 20120410 - Up until now -J (clear cache on login) was ignored if a pool was being used. 20120405 - Previously if your user ID's "maxdown" limit was too small (less than 2,000,000 * the number columns in your result) you would get an error even if the result set was small. - Corrected an issue with variable substitution when applied to a SQL query. 20120329 - Improvments made to the lenient option. 20120312 - More meaningful error messages have been added in several places. 20120302 - The '-k' option when used with a SAM pool will now cause the ID to be logged in with kill=yes instead of always using kill=possess. - When the append syntax += is used if the query result has no rows then the append transaction will be skipped to avoid a bug in older versions of the system. 20120216 - Tendo will now emit a warning to stderr when it is out of date. 20120213 - Updated the gateway shortcuts tendo recognizes to support the new versioning scheme. 20120126 - Resolved an issue where tables referenced in SQL queries submitted to tendo fail to be matched up with tables on 1010data's servers when the table was sufficiently similar to other tables. 20111215 - Resolved an issue with downloading results with more than 1000 columns. 20111123 - Resolved a bug when using a ^ in the -o argument without also using an @. - Made several corrections in the tendo-guide.pdf help document. - Tendo now uses HTTP keepalive packets to help prevent proxies from timing it out during long queries/savetables. 20110923 - Removed some limitations on the mixture of the -u, -g, -p and -s switches with the TENTENUID, TENTENGW, TENTENPW and TENTENSID environment variables. 20110818 - Fixed a bug which truncated the fractions of a second from "datehms24" columns when formatting was enabled. 20110808 - Fixed a bug which caused formatting to be ignored if the column had "width" and/or "dec" attributes but no "type" attribute. 20110803 - Fixed a bug that caused some negative numbers to contain an extra ',' before the '-' when formatting was enabled. 20110609 - Fixed a bug which prevented clearcache from being called when the memory high water mark was greater than 2GB - Fixed an issue with outputting csv files with string fields greater than 1024 characters. 20110428 - Added pooled ID support. 20110418 - When downloading data with formatting enabled floating point columns with no explicit format are now handled properly. 20110415 - Resolved an issue regarding saving new tables and inheriting permissions. - Fixed bug preventing 32-bit builds from being able to write files larger than 4GB on Linux. 20110321 - Resolved consistency issue between GUI and tendo regarding output of formatted columns. 20110228 - When errors are encountered the tendo application will more regularly exit with an error code. 20110124 - Additional wild cards have been added to file output syntax. - Resolved stability issue when reading SQL from a file. 20101111 - Resolved compatibility issue with legacy versions of Linux. 20101102 - More information is returned when inspecting a quick query. 20100927 - Added ability to use stored procedures. 20100908 - General stability fixes. Data Hauler Changelog 20220921 - Added M1 Mac build of datahauler for download (CT-145) NOTE: only works for Mac OS 12.3 or higher 20220725 - tenup update 20220627 - CSDK/SSO update 20220525 - Fixed Linux build, which was not usable due to an import error (CORE-6008) 20210618 - Added support for a new SSO authentication tag (CORE-5477). 20210318 - Added support for a new SSO authenticationt tag (CORE-5362). 20210305 - Fixed precision issue in data downloaded from datetime columns (CORE-5314). 20201002 - (linux, windows) fixed runtime error loading json module. (mac) fixed undefined _libiconv symbol error. 20200706 - updated digital signature certificate (windows) 20190529 - Initial build for the new DataBlazer suite installer. 20190306 - Added macOS support. 20190118 - The DATABASE_TYPE "tenten" can be specified to upload from a 1010data data source. - After tables are uploaded, a directory "queries" is created in DSTPATH. A quick query is created in this directory for each pair of tables that share primary/foreign key column(s) in the source database. The quick query links the smaller table into the larger table. If the tables are the same size, the table containing the foreign key column is the base table. If multiple queries are created for the same base table, they are combined into an additional query. For example, there are three tables "sales", "product", and "customer" in some source database. The largest table is "sales". The table "sales" has foreign key columns "SKU" and "loyalty". The table "product" has a primary key column "SKU". The table "customer" has a primary key column "loyalty". The following quick queries will be created to model these relationships: - "sales_product" links "product" into "sales" on the column "SKU" - "sales_customer" links "customer" into "sales" on the column "loyalty" - "sales_product_customer" does both of the above - The -k/--kill and -K/--possess options are available to login with kill=yes and kill=possess respectively. The default login mode for connecting with a single UID has changed from kill=possess to kill=no. The default login mode for connecting with a SAM pool has changed from kill=yes to kill=possess. - The --auth-retry option is available to retry a failed login some number of times. The --wait option is available to specify the duration between login attempts. Both AUTHRETRY and WAIT continue to default to 0. - Changed behavior and fixed bugs in the PostgreSQL clone. The "Database" field in the connection string CONNSTR will limit the clone to the specified database. The --include-sys option includes (a) the "postgres" database, and (b) the "information_schema" and "pg_catalog" table schemas from each database in the instance. In prior versions, the "Database" field and the --include-sys option did not affect the PostgreSQL clone. - Fixed a bug on Windows (runtime FileNotFoundError). 20180806 - Changed behavior for connecting with SAM pools. --pool indicates that up to GROUPMAX available UIDs will be used from GROUPNAME to upload tables in parallel. GROUPMAX defaults to 16 and can be set with --pool-max. - Users may specify "redshift" for DATABASE_TYPE to clone a database from a Redshift cluster. - Each directory and table created in a clone is given a title, by default, equal to the "basename" of its 1010data object path. For example, when cloning into DSTPATH "foo" from source database "bar", the directory "foo.bar" is created with the title "bar" and the table "foo.bar.baz" is created with title "baz". Table titles can no longer be specified on an individual basis using --passthrough. In prior versions, directories were created without titles and each table was created with a title equal to its full 1010data object path (tenup default) unless otherwise specified. - The --owner option is available to specify the owner of all objects created in a clone. Otherwise, the owner defaults to the user/group ID used to connect. - The --uploaders option is available to specify the users/groups that can upload into any of the directories created in a clone. The single argument UPLOADERS can be a comma-separated list of user/group IDs, "!" to inherit uploaders from DSTPATH, or "." to limit uploaders to the user/group ID used to connect. UPLOADERS defaults to "." (private). In prior DH versions, directories were created with uploaders equal to users (PERMISSIONS). 20180321 - Rebuilt with updated versions of CSDK and tenup.