Virtual memory

The quantity of data that your session can handle at the same time is determined by the amount of virtual memory assigned to your user ID.

The amount of virtual memory allocated to your 1010data Insights Platform session is displayed to the right of the session memory meter.

Figure: Session memory meter

As you perform your analyses in the Insights Platform, the meter reflects how much virtual memory you have used. You can think of this value as a current high-water mark.

There are limits on the size of a process's virtual memory (i.e., on the amount of data that the session can handle at once). If you work with relatively small tables, your process(es) use small amounts of virtual memory, and you should never even come close to the limit. But if you are dealing with large tables with hundreds of millions of rows, it is possible that you may run into the limit under certain circumstances. This qualification is important: Just because you are dealing with large tables doesn't mean that you will have a problem.

The Insights Platform is designed to process data piecewise, dealing with relatively small amounts of data at a time rather than all of it at once. But there are cases when it may be necessary to process large amounts of data at one time, either because there is no practical way to do it piecewise or because doing it piecewise would seriously impact performance.

Troubleshooting memory issues

If your current session starts to behave strangely (e.g., you see cryptic error messages) or stops responding altogether, it may be because you are approaching or have exceeded your virtual memory limit. After the virtual memory limit is surpassed, the system can behave unpredictably.
Note: If you see any message containing the word wsfull, it is almost certainly a sign that you have hit the virtual memory limit.

Even if you do run out of virtual memory, the system as a whole is unaffected. There is generally no impact on other users, and you won't affect any of the data on the system.

You always have the option of logging in again. That will end your first session and set things right. But it is sometimes possible to recover without losing the session. Also, even if you start a new session, you will have to do things a bit differently the second time around; otherwise, you will end up in the same place.

Here are some steps we recommend:

  • Clear the cache

    First, see if you can recover without logging in again by clearing the cache. See Clear the cache for details.

    If the virtual memory value goes down, your session is OK; if it is still too high, you will have to log in again.
    Note: Remember to save your work before starting a new session. 

  • Restructure your query

    There is usually more than one way to do an analysis in the Insights Platform, and some methods use significantly less memory (and run more quickly) than others.

Unfortunately, it is essentially impossible to monitor virtual memory usage at every step in a process or to predict how much virtual memory a given query will use. The only way to ensure that you won't hit the limit would be to impose significant restrictions on all queries. However, such restrictions would end up disallowing many viable queries as well. Rather than imposing such arbitrary restrictions, we put the full power of the Insights Platform under your control. This gives you the flexibility to run a greater variety of analyses using different methods to get the results you need using the most efficient queries possible. The downside, of course, is that you can encounter virtual memory problems; but, as noted above, you can recover and you cannot affect anyone else. However, keeping an eye on your virtual memory usage can help you steer clear of the limit before hitting it.