Workspace

Your workspace is the amount of virtual memory that has been allocated to your 1010data session.

The size of your workspace is displayed in the top right corner of the 1010data web interface:

As you perform your analyses in 1010data, your workspace will be increased to reflect how much virtual memory you have used. You can think of this workspace value as a current high-water mark.

There are limits on the size of a process's virtual memory (i.e., on the amount of data that the 1010data session can handle at once). If you work with relatively small tables, your process(es) use small amounts of virtual memory, and you should never even come close to the limit. But if you are dealing with large tables with hundreds of millions of rows, it is possible that you may run into the limit under certain circumstances. This qualification is important: Just because you are dealing with large tables doesn't mean that you will have a problem.

1010data is designed to process data piecewise, dealing with relatively small amounts at a time rather than all of it at once. But there are cases when it may be necessary to process large amounts of data at one time, either because there is no practical way to do it piecewise or because doing it piecewise would seriously impact performance.

Troubleshooting Workspace Issues

If your current session starts to behave strangely (e.g., you see cryptic error messages) or stops responding altogether, it may be because you are approaching or have exceeded your virtual memory limit. Once the virtual memory limit is surpassed, the system can behave unpredictably.
Note: If you see any message containing the word wsfull, it is almost certainly a sign that you have hit the virtual memory limit.

Even if you do run out of virtual memory, the system as a whole is unaffected. There is generally no impact on other users, and you won't affect any of the data on the system. So, don't worry about crashing the system or anything as catastrophic as that.

You always have the option of logging in again. That will end your first session and set things right. But it is sometimes possible to recover without losing the session. Also, even if you start a new session, you will have to do things a bit differently the second time around; otherwise, you will end up in the same place.

Here are some steps we recommend:
  • Clear the cache

    First, see if you can recover without logging in again by clearing the cache. See Clear the cache for details.

    If the workspace value goes down, your session is OK; if it is still too high, you will have to log in again.
    Note: Remember to save your work before starting a new session. Save it as a Quick Query or save your actions as a text file. Or, if you are in the Edit Actions (XML) dialog, copy the XML code and paste it into a text editor.
  • Restructure your query

    There is usually more than one way to do an analysis in 1010data, and some methods use significantly less memory (and run more quickly) than others.

    See Writing Efficient Queries for suggestions.

  • Change your settings

    If you've tried clearing the cache and restructuring your queries, and you believe you're still exceeding the virtual memory limit, click Actions > Advanced and change some of the settings that control memory utilization.

Unfortunately, it is essentially impossible to monitor virtual memory usage at every step in a process or to predict how much virtual memory a given query will use. The only way to ensure that you won't hit the limit would be to impose significant restrictions on all queries. However, such restrictions would end up disallowing many viable queries as well. Rather than imposing such arbitrary restrictions, we put the full power of 1010data under your control. This gives you the flexibility to run a greater variety of analyses using different methods to get the results you need using the most efficient queries possible. The downside, of course, is that you can encounter virtual memory problems; but, as noted above, you can recover and you cannot affect anyone else. However, keeping an eye on your workspace usage can help you steer clear of the virtual memory limit before hitting it.